Instrumentation

Development

Instrument Design

I have a number of ideas for new instruments bouncing around in my head, but funding for new instrument development is difficult to find. I have been involved, though, in the design of one new instrument, a capacitive wave gauge. Now I don't think I'll win the Nobel prize for my contribution to this endeavor, but a cursory examination of the design elements might be useful.

The sensor for the APL wave gauge system is an anodized tantalum wire. While tantalum is a metal, and hence a good conductor, the anodization process forms a very thin layer of tantalum oxide about the wire's exterior. Tantalum oxide is a good insulator and dielectric, so the sensor works by measuring the capacitance between the metal interior of the wire and the surrounding sea water.

The original idea for a tantalum sensor was developed by Blythe Hughes and Ron Chappell of the Defence Research Establishment in Vancouver, British Columbia. Their original system utilized a sea water anodization process which resulted in a rather irregular and brittle oxide coating. An engineer at APL, Chris Keller, further developed the system by switching to anodizing in a weak acid solution. This made for more uniform coatings. Keller also modified a circuit originally used by the Canadians to convert the capacitance of the improved sensor into a voltage that could be fed into a data acquisition system.

After Keller's retirement in 1990, I took up the development system and together with my colleagues Frank Monaldo and Jim Allison performed an extensive series of experiments to quantify and understand the anodization process. We uncovered undesirable artifacts in the sensor response and developed methods to minimize those artifacts.

At the same time, I began the design of a new, lower power version of the wave gauge electronics board. I came up with a brand new design that promised lower cost, lower noise and lower power operation. I breadboarded a simplified version of the design and verified its operation. I then gave the full design to one our electronic engineers, Robert Miller. Bob spent a few minutes looking at the design and then proceeded to lecture me for 15 minutes on all of the design mistakes that I had made. The op amps weren't bypassed properly and would likely oscillate, the oscillator was only marginally stable and would likely have startup problems, the dependency on power and temperature fluctuations were not what they could be, and the circuit would fail catastrophically if the sensor was momentarily shorted. Instead of being angry, this just reinforced my view that Bob was the right guy to finish the design. He cleaned up the design, had some printed circuit boards made, and tested the final assembly. The system ended up cheap, precise, and rugged; a tough combination to beat.

(As an aside to this story, I'll point out that I actually have a Master's Degree in Electrical Engineering from a name-brand university, while Bob does not. This just goes to show that a piece of paper hanging on the wall signifies little!)

A similar story can be told about the mechanical design, where I came up with an initial concept, showed it to a mechanical engineer who then improved upon it. The moral here is that no one person can do these things alone. If you are going to design an instrument, be sure to use some experts along the way, if for no other reason then to validate your design. Working with others will make it no less your instrument, because instrument development always requires someone to see the system through to the end. But working with others will significantly increase your chances of success.

The final act of developing the new wave gauge system was to test the entire system. In these tests, we built several copies of the system and tested the design's tolerance to expected component variations. We measured the power consumption of the units as a function of voltage. I tested the stability of the unit as a function of voltage, both in the middle and the extremes of its ranges. I cranked the voltage up on one unit until it failed, measuring its failure point, and its failure mechanism. This let me know which parts to provide spares for in the field. I tested the unit stability versus temperature and water salinity. We checked the linearity of dozens of sensors and tested the sensors and electronics for short and long-term drift. In short, I did everything I know of to calibrate and test the system and all of its components. I documented all of these tests so that now we have a relatively good understanding of the system as a whole. And this was all done for a simple wave gauge!

Buoy development is in many ways similar to the development of an instrument. Here the stress is placed on the system, both the mechanical aspects of holding the instruments, but also the electrical and electronic aspects of power distribution, data multiplexing, storage and transmission and instrument control. The system is the key, and how well the parts merge to form the system will determine the success or failure of the buoy.

I can't provide you with a complete set of guidelines for designing buoy systems. That would take a longer book than this and more expertise than I possess. I can tell you though that the same advice that I gave above for instruments, also applies here. Get experts in the various fields involved early in the design. Test, calibrate and characterize every aspect of the system that you can get your hands on. Finally, you should design for every eventuality, anticipate failure modes and build in protection against them. To illustrate these latter points, I'll provide an example of how not to build a buoy system.

How NOT to Build a Buoy

On a recent cruise, a new buoy was deployed for the first time by a colleague who had never designed a buoy before. When I first saw the buoy I knew we were in for trouble. The buoy was a free-floating surface-riding device whose primary sensor was an optical device. As anyone who has ever worked in an optical laboratory knows, all optical devices are painted flat black to cut down on stray reflections. Well the designer had really taken this convention to heart. His entire buoy was painted flat black! To make matters worse, the buoy was only about 2 m high with less than 1 m above the water line. I pointed out to the designer that we would be deploying this buoy near or possibly inside the Gulf Stream and that it would be difficult to find if it got away from us. He assured me that this was not going to be a problem because it was quite easy to see.

On the first day of deployment of the new buoy, things went from bad to worse. The buoy was rigged and lowered in to the calm waters of the Atlantic. Its gross ballasting appealed to be fine, so the designer gave the signal to move the ship away from the buoy. We had not traveled 50 m away when the buoy keeled over and layed down on its side. We circled back, hooked onto the buoy, pulled it upright and then again released it. Again the buoy fell over. The designer came to the amazing conclusion that the boat must be sucking the buoy over, so he instructed the skipper to try again but this time with minimum use of thrust. This time the buoy stayed up for a couple of minutes, but again it fell over.

It was clear to all of the rest of us on board that the buoy just wasn't stable. I asked the designer who had decided upon the distribution of buoyancy and weight. He replied that he had. I then asked about the results of the stability calculations for the buoy. He didn't quite understand the question, replying that it floated fine in a lake. It was at this point that I understood that he had done no stability calculations. Appalled, I walked away.

After several days of screwing around with the buoy, I finally convinced the designer to add some extra weight to the bottom of the structure. I wanted to add the weight on a line, so that the righting force would be maintained even if the buoy started to lie over, but he felt no need to do that and just attached it to the bottom of the buoy structure. This time the buoy rode a bit lower in the water when it was deployed, but it seemed to stay upright as we sailed away to deploy additional instruments.

Several hours later someone came in to the lab to inform me that the buoy had been lost. Apparently the designer of the buoy left the bridge for some time to attend to some data analysis and so had lost the buoy both visually and on radar. We knew where we had dropped the buoy off, so I asked to see the notes on the track that the buoy had taken while it was in the water. The designer had recorded its distance from the ship as a function of time while he had tracked it, but not its direction! Thus we had no track to project for the search. Still, we recovered our other instruments and proceeded to search for the lost buoy. We were joined in our search by another research vessel and a P3 aircraft that were also participating in our test. The costs of the search were quite high, but we felt that we had no choice.

A couple of hours into the search I innocently, and in all seriousness, commented that it would be dark soon making the buoy easier to find. The designer was surprised and asked me why I would think such a thing. I uneasily explained that when it got dark, the beacon light would come on and be easily seen at considerable distance. He then asked, What beacon light? I was stunned. Here is a guy that built a small buoy, painted it flat black and deployed it near the Gulf Stream edge with no beacon light and no Argos transmitter!

In what must have been the act of a kind and benevolent supreme being, the P3 finally located the buoy. It was lying on its side several kilometers from our location. When we recovered the buoy I examined it. At this point I was not at all surprised to find that the designer had not even written his or his institution's name and address on the buoy.

We finished the cruise without further incident. In retrospect I'm the first to admit that the designer of this buoy is a bright guy. I am certain that he learned a lot of lessons for his next cruise. Still, his mistakes should have been avoided by consulting with any number of people who have experience in buoy design. The sheer number of mistakes that he made with this single system was astounding. It was clear that he did not understand the environment that the buoy would be operating in, and that he had not given any thought to the recovery of his system.

Design Rules

The best path to good system design is to believe in Murphy's Law: If anything can go wrong, it will. As I will say time and time again in this book, you should expect things to fail when you go to sea. The trick is to design a system that can continue to work if something fails. To understand the philosophy you need to understand the concept of point failures. Let me illustrate with the simple example of power distribution on the APL wave spar buoy.

The power distribution system in the APL wave spar buoy begins with a single set of 12-volt batteries, wired in parallel. The power from these batteries is then fed through a remote control system to each of the individual instruments. The remote control system allows each instrument to be turned on or off individually.

Beginning with this simple conceptual design we then considered different failures that could occur and how each would affect the operation of the system as a whole. For example, suppose one of the instruments flooded and shorted out. While the remote control system was designed to turn off individual instruments, the batteries could be drained if we didn't notice the problem right away. Thus we put fuses in line with each individual instrument. We reasoned that if the fuse blew, we'd lose the use of the instrument but save the system.

So far, the system is still quite simple. Next, we considered the effect of a failure in the remote control system. That system could fail with the power left on or off or in some undetermined state. We decided to run the power for each instrument through a set of toggle switches which allow us to manually turn the instrument on, off or set to remote control. These switches were mounted on the outside of the telemetry and control box, where they can be reached from a small boat in an emergency.

Then we considered what would happen if the main power line from the batteries was shorted accidentally during work on the system. Our concern that we could burn out a cable or connector led us to add a fuse to the main power line. This fuse was made larger than any of the fuses for the individual instruments so that it would blow last. It is important to realize that this fuse is a new single point failure mode for the system - if it fails, the entire system fails. In this case we made the conscious decision that if this failure mode occurs, the system is in deep, deep trouble and in need of serious repair.

Next, we worried about the possibility of the batteries, which are 10 m underwater in our design, running out of power before the end of an experiment. Thus we added a second power cable running up to the surface from the batteries to allow for charging of the batteries while the buoy is deployed. This has, in fact, been done using an umbilical cable from a nearby R/V. The charging cable was routed directly to the batteries, instead of through the control box, to prevent the voltage drop across the extra 10 m of cable from raising the voltage at the control box to dangerous levels. Finally, we added voltage monitoring circuitry and firmware to the control system to monitor the battery state.

In this real-life design case, we first developed a simple conceptual model that provided the functionality that we needed. Then we spent considerable effort analyzing this simple design to uncover what could fail in the design and how each failure would affect the system, giving the highest priority to single point failures that could disable the whole system. We then iteratively modified the design to anticipate and abate the worst failures. This is the only approach to design that I know of in the face of Murphy's Law.

Preparation

I cannot tell you how many times that I have heard people discuss problems that they had at sea because they had not checked out their system before sailing. Preparation for a sea cruise is hectic and invariably undertaken without sufficient time to check everything out. Time and time again I have seen systems wired together for the first time the day before we set sail and software being written on the transit out to the experiment site. Sometimes, due to delivery schedules of components, this is unavoidable, but in other cases it is a sign of unprofessional preparation. It should always be the course of last resort.

Let me give an example. On my first cruise I was the lowest form of life at sea: a graduate student. I was on a ship from a well-known institution simply for the at-sea experience. I had finagled my way onto the cruise because I figured that I should go to sea at least once if I was going to call myself an oceanographer. Doron Nof, a well-known theoretician who was my major professor, disagreed. His initial view was that there was nothing in oceanography that couldn't be learned from behind his desk. Naturally, I am exaggerating here, but I wanted to try a cruise just once and I talked him into supporting my adventure.

Once on the ship, I was determined to make myself useful and to learn something. I attached myself to the chief engineer in the scientific party, offering to help in his work of setting up and checking out the equipment. For his part I think he accepted my invitation so that he'd have someone to fetch him coffee. To me it seemed a fair bargain.

On our second day at sea, the engineer asked me to help him change over the CTD to a new winch with brand new cable. The old cable had been in service for several years and was at the end of its safe operating life. The new cable had been ordered from the same manufacturer that had made the original. Unfortunately it was delivered late to the ship so that the engineer only had time to spool it onto the backup winch prior to our setting sail. To switch to the new system we had to terminate the new cable, check the entire system out electrically and then jury rig a new slip ring assembly onto the backup winch. The slip ring assembly went together easily as the engineer had thought ahead and brought sufficient parts to do the job. The cable termination appeared to go well. The termination involved pouring molten lead into a mold to provide a strong mechanical connection. The trick in the process was to avoid damaging the insulation of the internal electrical conductors when the molten lead was poured.

When the termination was complete, we hooked up the deck box, which controlled the Nisken bottle releases, to the slip ring assembly and the release to the newly terminated end. The system didn't work. The engineer patiently explained that sometimes, as careful as you may be, the internal wires get damaged by the termination process. He cut off the termination with a hacksaw, and spent another half hour re- terminating the cable. Again the system didn't work. This time we decided to check out the cable for continuity and short circuits. It checked out fine. We then hooked up the deck box and release, and examined the signals at both ends of the wire. The signal at the release end was very weak, despite the good continuity check. By this time there was a lot of cussing and scratching of heads to try to figure out what was going on.

I got the engineer to explain to me how the deck box system worked. The system utilized audio tones, like in a computer modem, to control the instruments. The deck box basically generated these tones under user control and sent them down the line using an AC current source. I suggested that maybe the electrical capacitance of the cable was too high for the deck box to drive the signal down the line. The engineer explained to me in patient tones that the cable was made to a particular capacitance specification and so this could not be the problem. He added that we didn't have a capacitance meter on board so we couldn't check the system even if we wanted. Despite his certainty that the problem lay elsewhere, he was in the end unable to figure out how to proceed, so he let me work on the system.

I computed the total expected capacitance of the cable from its specifications. I then got a comparable size capacitor from the ship's stores and a variety of resistors to create a long time constant. I hooked up a resistor in series with the capacitor and measured its resistance using an analog multimeter. As the capacitor charged up in the circuit, the multimeter swung from a low resistance reading to infinity. I changed resistors until the time constant of the needle swing could be easily measured. I associated this time constant with the particular capacitance I had selected. When we hooked the same resistor in series with the cable and placed the multimeter on it, the needle swung much more slowly to infinite resistance. By adjusting resistors to arrive at the same time constant as in the test case I was able to show that the capacitance of the cable was about three times the specification.

At this point we had figured out what the problem was. The solution then followed. At my suggestion the engineer got out the schematics for the deck boxes. The decision was made to parallel the output stages of two deck boxes to double the total drive current. When he hooked up this kludge of a system to the slip ring assembly everything worked fine. The cruise proceeded from that point with out major incident.

If we had not been able to diagnose and fix the problem with the cable, we would have had to scrub the entire multi-million dollar experiment, or taken a major risk in losing $100K worth of CTD by switching back to the old, worn cable. While it would have been easy to diagnose and fix this problem on shore, we barely managed to avoid these rather dire consequences at sea. Most times things won't work out this well, so it pays to fully check out your instruments prior to sailing.

In a less dramatic example, I once took delivery of some current meter mooring frames on the dock just before I was to set sail on a Russian research vessel. The current meters are supposed to bolt into these frames, which then hold the current meters securely on the mooring. On our second day at sea I was preparing the moorings for deployment when I realized that not only did the frames not come with any bolts, but that I had failed to bring along any bolts of sufficient length to mount the current meters into the frames. After much cussing I asked my Russian colleagues if they had any long bolts. When they told me no, I cussed for a while longer than sat down on the deck staring blankly at the useless frames. Naturally I felt like an idiot.

Fortunately for my reputation, this story too had a happy ending. One of the Russian technicians noticed my predicament, motioned for me to wait, and headed off to the machine shop. Not being known for my ability to follow instructions, I picked myself up off the deck and followed him to see what he was going to do. In the ship's small machine shop, he got some hexagonal stock, put it on a lathe, and proceeded to make the necessary bolts. I, like most scientists that I know, thought that bolts came from a hardware store. I never realized that they could be made by real people! The technician's ingenuity saved the day and we deployed the moorings on time. After deployment, I went into our lab, fired up my computer, and added several pieces of assorted threaded rod and nuts to my list of parts and material to take on each experiment. I am determined not to make the same mistake again.

My primary lesson from this was to assemble and check out everything possible before going to sea. When I am going to deploy an instrument (or buoy or mooring), I first completely assemble the instrument in the lab. Then I check out its operation. Finally I take the system apart for shipment, being careful to bag and mark all of the nuts and bolts, cables, and odds and ends necessary to make the system work. This doesn't insure that you will have remembered everything, but it does seem to help.

One case where this procedure didn't quite work out occurred on an experiment in 1989. I was chief scientist on the experiment, which was land-based, and someone else from my lab, acting as chief engineer, was responsible for the instrumentation. We were to deploy a spar buoy as our major instrumentation platform during this experiment. I was working on the software for the system when one of the technicians came in to tell me of an argument on the dock. I went out to find my chief engineer arguing with another engineer involved in the design of the spar about how much flotation to bolt onto the structure.

This was a serious argument. A spar buoy is a long vertical pole with flotation in the middle and a weight at the bottom. The amount of flotation is finely adjusted so that the spar will float vertically in the water, with a fixed amount of the pole sticking up out of the water. The excess pole sticking out of the water acts as reserve buoyancy for the system, balancing any tendency for the spar to sink. For example, the APL wave spar my engineers were arguing over, was 12 m long and weighed over 900 kg in air, but had a reserve buoyancy of just 10 kg per meter of submergence. If enough buoyancy is not placed on the spar when it was deployed, then it would sink to the bottom. This particular spar carried about $100K worth of instruments, so its loss would have been rather substantial.

I tried to calm the argument by suggesting that we could calculate how much buoyancy was needed with a set of careful measurements. But we first needed to know how much the spar weighed in air. I was appalled to find out that nobody could tell me. Without an accurate weight, there was no way to make any kind of estimate. I decided on the spot to add every piece of buoyancy that we had brought along. I had the smallest pieces of buoyancy attached using tie wraps so that a diver could cut them off easily if there proved to be too much buoyancy. I figured we could use the removable buoyancy and add weights to balance the structure after it was placed in the water.

Confidant that we had enough buoyancy on the spar, we put it in the water at the pier and towed it several kilometers to the test site. I sent out a technician and a diver in a small boat with the spar to check its buoyancy. When they got 10 m away from the boat I yelled across to the technician to let the spar float, but not to let go of the line we had attached to the spar. He heard the first part correctly, but missed the word 'not' in the second part. As he tossed the line over the side of the zodiac, both the spar and my heart began to sink. The spar's natural period of oscillation is about 16 seconds so its downward motion was agonizingly slow. I yelled until I was hoarse, and our technician just managed to maneuver the zodiac around and grab hold of the spar before it slipped beneath the surface. Actually I think the spare buoyancy in the telemetry box at the top of the spar would have prevented it from sinking, but it came awfully close. In the end we had to borrow some additional floats from the small work boat we were on to complete the deployment.

Ever since this episode, I personally have assembled and weighed the spar back at our lab prior to shipment. I have measured the volume of every piece on that spar and can estimate within a kilogram or two how much buoyancy we need to make it float at the proper level.

From this experience, along with others, I now am careful to distinguish between the delegation of work and the delegation of responsibility. As test scientist I was responsible for coming back with the data. Our sponsor didn't want to hear that we had lost a buoy because of a mistake made by one of our engineers. He would not have cared that it wasn't my fault. Since then I have learned that one must delegate the work, but that the responsibility lies with the scientist. For that reason, I either do critical pre-test checks myself, or I rely on people that I can absolutely trust. I try to leave nothing to chance.

The Effects of Poor Preparation

Finally, I'll relate a story that did not end happily. As a favor to a friend, I signed up for a cruise in the Pacific. My friend was the chief scientist for the overall test and was concerned about one of the ships involved, which was to make environmental measurements. The problem was that there were no APL scientists on board, just technicians and engineers. I signed on as an environmental consultant just to help out my friend.

A primary instrument on this cruise was a downward looking Acoustic Doppler Current Profiler (ADCP). It was to be operated by an engineer who had previous experience with such systems. I arrived at the dock just two days before we were to sail. The instruments had already been set up on the ship and a few-hour long shakedown cruise had been completed to test the instruments. I was told that the ADCP had been fully tested and was working just fine.

We set sail for an area several hundred kilometers from Hawaii under rough conditions. Not one for heavy seas, I was sick in my bed for the first three days. When I finally could get up and about I checked to see how the instruments were working. Everything was working fine but the ADCP.

It seemed that everyone but myself and the engineer responsible for the ADCP had shown up for the onload and installation a week prior to sailing. I was late because of another test, but the engineer had been vacationing on the islands. By the time he finally decided to show up, the technicians were mad about having to do his work. Instead of facing the constant wrath of the technicians, the engineer decided that he'd install the system himself.

Once I had reconstructed this history from the participants it took us a few hours of work before we were finally able to determine that the engineer had miswired the stepper input from the directional gyro. When we rewired the system correctly it began to work. To my way of thinking we lost several days of data because of this engineer's lack of preparation and dedication to his work. I was most bothered by the fact that he had assured me that the system was working fine and that the shakedown cruise data looked good. It was only later that I found out that he hadn't processed the data completely when he gave me this report. It was for that lie that I have sworn not to go to sea with him again.

An Example Preparation Checklist

Preparation of your instruments is one of the keys to success at sea. This lesson cannot be overstressed. Naturally each instrument is different, but as an example, here are the procedures I now go through to deploy our APL wave spar buoy with 3 wave gauges, current meters and meteorological system.

Wave Gauges

The buoy system uses three wave gauges designed and built at APL. These system use a homebrew anodized tantalum wire sensor which has proven to be sensitive, extremely linear, stable and rugged. The capacitance of the sensors, which is linearly related to water level, is measured and converted into a voltage within a small electronics assembly.

1. Though each wave gauge uses only a single wire sensor, I typically take a dozen or so sensors on each cruise, in case the sensors are broken or damaged in some way. We manufacture each capacitive sensor using a simple anodization process. The sensors are cleaned, aged in a salt-water solution, and then calibrated. The calibration values are saved in a notebook that is kept at the lab, and a copy of the calibration is included in the bag where the sensor is stored prior to use.
2. Though each wave gauge uses only a single electronics assembly, I usually prepare four or five units for the field. Each unit is cleaned when removed from storage. Each electronics package is powered up and tested for proper operation. An input current as a function of operating voltage is measured for each unit to look for failures and to test the unit's stability. I have known buoy system's to fail due to excessive current draw from a single instrument, so I always check the current draw of every instrument on the buoy.
3. The seals on each assembly are checked and resealed if questionable.
4. Finally the wave gauges are each calibrated and adjusted so that their capacitance-to-voltage transfer functions are linear and nearly identical. These like-calibrations make their replacement in the field easier. All calibration data are stored. When questions arise during, or after a test, it is then easy to perform post-test calibrations on a particular instrument and meaningfully compare these with the pre-test calibrations. Good instruments don't drift.

Current Meters

The spar buoy can accommodate up to three current meter / CTD packages. We use a UCM-40 Mk II current meter manufactured by NE Sensortec of Norway. These units are rugged, low power three-axis acoustic current meters with a resolution of 1 mm/s with 1-second integration time. All communications with the units are provided via an RS-232 link to a controlling computer.

1. Retrieve current meters from storage and clean. All units are powered up and tested for coarse operation in a bucket of water. Operating current versus operating voltage is measured and compared with specified values.
2. Each unit is taken apart and the internal clock battery voltage is checked. The batteries are replaced if necessary. O-rings are cleaned and regreased before reassembling the units.
3. The temperature and conductivity cells in each unit are calibrated using a stable bath and a conductivity and temperature reference system that we have at the lab. Adjustments are made to internally stored coefficients to bring the units into specification. Past current calibrations, using a stable flow tank and a laser Doppler velocimeter for reference, have shown that the units are stable with time, so this more complex test is not performed each time.
4. The setup parameters for each instrument are planned and written down. Each instrument is powered up and setup for their operation in the field. All of the status and setup information is then downloaded from the instruments and recorded into a file for future reference. This can be very useful in insuring that the operator has not changed the settings accidentally. From this point on, all communications with the instruments are recorded. This setup includes measurement timing, averaging lengths, designation of channels to be reported, setting of magnetic deviation in the area of the test, and a variety of other instrument specific variables.

Meteorological Systems

1. System is retrieved from storage and cleaned. All anemometer bearings are checked and replaced if necessary.
2. The system is powered up and current is measured. Each sensor is checked for proper operation, including temperature, humidity, wind speed, direction, and compass. Detailed calibrations of each sensor are performed.

Wave Spar Control Computer/Telemetry System

All instruments on the spar are connected to a homebrew control computer and telemetry system. This system allows the user on a nearby ship to control each individual instrument, and read data from the instruments in real time. The system can also be programmed to serve in a store and forward mode where the data from the instruments are stored for later forwarding to the ship. This system includes the control computer, a spread spectrum RF telemetry system, a remote controlled instrument power distribution system, and cabling.

1. The control computer and telemetry system are powered up and tested. A backup external direct connection to the computer (in case the telemetry system fails) is tested. Input current for the entire system with and without telemetry system is measured.
2. The remote control power distribution system is tested by turning on and off each and every system while checking each connector for the proper DC voltages.
3. The entire system, with all instruments and cabling, is wired together in the laboratory for testing. Each instrument is tested individually. All of the instruments are then tested in unison. The store and forward capabilities are tested. Typically an overnight test is then run and analyzed for any discrepancies or data glitches that could represent system problems. Any new features that have been added to the control firmware are given special attention. The system is powered on and off repeatedly to insure that all of the instruments and control computer start up correctly. Total system current is measured and compared with expected system totals.

Spar System

1. The entire spare system is then assembled, including all hardware, instruments and cables. The battery systems are charged and discharged to measure their capacity, replaced if necessary, and then recharged. All hardware is checked to insure that it is painted or anodized aluminum or stainless steel. The system is checked for operation and all cabling is dressed with tie wraps and duct tape (the oceanographer's friends). The entire system with and without flotation is then weighed. The individual flotation elements are weighed.
2. When the system is completely ready to go and fully operational, the area is cleaned of all debris and parts. An empty table is set up nearby to accommodate the tools and parts during the disassembly. The spar is then carefully disassembled with each set of screws, nuts and bolts going into labeled zip lock bags. All of the hardware, instruments, cabling, and bags of fasteners go into the shipping boxes. This procedure insures that when the boxes are opened for deployment that all of the parts will be present for the proper assembly of the spar.
3. Finally a deck box containing spare hardware and fasteners along with expendables like tie wraps and tape is prepared. This is shipped along with the spar just in case we missed something or in case we decide to change the spar configuration at the last minute. I always take spare instrumentation support arms into the field in case we decide to add something. I have even designed spare unused channels into the control computer for such an eventuality. I don't forget to take an assortment of spare underwater connectors and cables, along with a wide variety of other stuff that just might come in handy.
4. Finally, all of the documentation for the spar including mechanical drawings, electrical and electronic diagrams and schematics, connector wiring diagrams, firmware and communication documentation, power measurements, weights and volumes charts, and instrument calibrations are bound together into a single notebook. Multiple copies of some drawings are included just to insure that the information is still available if a document is removed from the book in the field and lost. Included with this book is a mosaic of photos taken of the entire assembled spar showing the location of each and every part and bracket. These photos, taken just prior to disassembly, make the reassembly of the system that much easier. They also come in handy two years down the road when you need to reassemble the system after it has been in storage.

While this checklist is quite specific, a more general checklist for instrument preparation is easily constructed:

• Completely test, characterize and calibrate every piece of equipment
• Check power draw and thermal sensitivity against design specs
• Assemble and test complete systems
• Test for periods comparable to deployment, if at all possible
• Check all batteries and replace if questionable
• Keep all parts together and labelled when packing
• Pack all required documentation

Know Your Instrument

There are few absolute rules in this book, but this is one of them. A complete understanding of the operation, care and feeding of your instruments is a must for successful operations. If all goes well, your knowledge should not be needed, but things rarely go well at sea. When conditions change, things break, or just don't work like you planned, your knowledge of your instruments may well determine your ultimate success or failure.

In 1989 I was involved in a collaborative effort with Dr. Guy Meadows of the University of Michigan to measure the surface currents in the wake of a large ship. While this is not an easy task, our approach was a simple one. We decided to seed the wake with small drifting flares. The location of these drifters would then be tracked using a video camera mounted on a helicopter hovering over the wake. Because I was already planning on using video to image the ship wake for other reasons this was a natural extension of my efforts. The only necessary addition to my planned system was a means of measuring the pointing direction of the camera. (I already had made provisions for recording altitude.)

After asking around, one of my colleagues at APL recommended that I use a vertical reference gyro to measure the helicopter roll and pitch. Best of all, he knew where we could borrow one. I made all the arrangements and was soon the proud proprietor of a vertical reference gyro. The owner had thoughtfully provided us with wiring schematics for the gyro, so I passed the whole mess to a technician who was working with me on the project. I explained to the technician what I wanted and asked him to find the correct power sources for the gyro and to wire it into the system.

A few days later I got a call from the technician. It seems that this particular gyro required an inordinate number of special voltages. The technician had managed to find the required 28 volts DC and the required 115 volts 400 Hz AC, but he didn't have an easy solution for a 12 V DC input. I suggested using a small DC-DC converter as the current was not great, but time was running out and we had a million other things to get ready before the test. The technician suggested that maybe we didn't really need this voltage after all. It turned out that he had powered the system up without the 12 V supply and it seemed to work fine. Furthermore he had taken the gyro apart and had traced the 12 V line to a switch that was activated when the gyro was in the vertical position and a small sealed block that he took to be a resistor. He had come to the conclusion that this 12 V input was actually an indicator of when the gyro was vertical. It all made sense to me, so I agreed that we should just skip the 12 V power supply.

Two weeks later, the system was installed in a helicopter in California and we tested it out. Everything worked fine on our first, short test flight. The next day the experiment began. About a half an hour into the measurements, I noticed that the aircraft pitch and roll looked funny. As I watched, the computer indicated that the helicopter rolled onto its side and stayed there. I was having no trouble sitting on the floor of the helicopter and so I knew there was something wrong with the instrument. I powered the system down, latching the gyro into the vertical position. Then I powered the gyro back up, and unlatched it. Over the next half hour the gyro again drifted so that it indicated the helicopter was flying on its side. No matter what I did, I could not get the gyro to work correctly.

When we landed we checked out the whole system and it seemed to work fine. The gyro only had problems in the air. Furthermore, we had no spares (another no-no) and could locate no one that seemed to know anything about this particular gyro. We struggled with the system for two weeks, but in the end, the portion of the experiment that required this gyro was lost.

When I returned to APL I decided to do some reading about vertical reference gyros to try to figure out what went wrong. I read that a vertical reference gyro is simply a gyroscope that is set in a dual-gimbaled mount. The gyro is spun up with the mount locked so the gyro shaft is vertical. Only when the gyro reaches full speed are the gimbals released. Sensors on the gimbals then measure the angle of the gyro shaft, which remains vertical, with respect to the case that is bolted to the aircraft. That is at least how an ideal vertical reference gyro works. In the real world of course there is friction and so a real gyro will slowly rotate away from vertical, or fall over, unless it is somehow occasionally forced back to vertical. The scheme for doing this involves a switch that is thrown when the shaft is nearly vertical and small righting magnets that give the gyro a small tug as it passes through vertical. This scheme, which relies on the fact that on average the aircraft has zero pitch and roll, acts as a low pass filter to keep the gyro erect. That damn 12 V supply which we didn't bother hooking up was the power for the righting circuit. Without it, as soon as the aircraft performed any maneuvers to bring the system out of level, the gyro would begin its slow fall.

It was an expensive lesson. We came back empty handed and I couldn't blame Guy if he decided to never work with me again. But it was a valuable lesson and I have taken it to heart. Now I try to learn as much as I can about every instrument that I use. I study the principles of operation. I read and reread the operations manual. I practice setting the instruments up in the lab. I do everything I can to insure that the instrument will work in the field, and that if it fails, that I know enough to either fix it or at least diagnose the problem.

Knowing your instrument may also prevent mistakes from being made in the interpretation of data. No instrument is perfect. They all have limitations and ranges of validity. Scientists who believe everything that their instruments tell them will typically have short careers.

A few years ago I was involved in an experiment off the Eastern coast of Scotland in the Sound of Sleat. The experiment involved several moorings, but this was the first time we had operated in this region. Realizing the risk of mooring in a new area, the chief scientist wisely deployed a bottom-mounted ADCP in the area about two weeks prior to the deployment of the moorings. When the ADCP was recovered prior to the experiment, it could provide us valuable information about the currents in the area that might be used to improve our moorings.

When the ADCP was recovered, the data were dumped and analyzed by a young scientist who was responsible for the instrument. When I asked him about the results he explained that the peak currents were 5 cm/s or less throughout most of the water column, but that the near-surface currents got extremely large, often exceeding 1 m/s! These results surprised me. I asked him how close to the surface these strong currents were observed. When he told me that these extreme currents were limited to the upper 5 to 10 m of the water column in an area where the total water depth was 80 m, I understood exactly where he had made his mistake.

ADCPs work by sending out beams of acoustic energy and listening for the backscattered energy along these beams from small bubbles or particulates in the water. The Doppler shift of the backscattered energy gives a measure of the fluid velocity towards or away from the ADCP. A typical unit will send out four beams, in a so-called Janus configuration. The four radial velocity measurements are then combined to estimate the horizontal and vertical components of the velocity.

In real life, the beams produced by the ADCP are not perfectly collimated. Because of limitations in the size and design of the acoustic transducers, the beam pattern has sidelobes, which contain energy propagating in directions other then the desired one. While the sidelobes are very weak, so is the scattering from particulates in the water. As long as the main beam signal is stronger than the sidelobe signal, the system works. When the beams from a bottom-mounted ADCP approach the surface, the sidelobe reflection from the surface becomes stronger than the backscatter from the main beam and the ADCP readings are contaminated. [For a detailed description of the process see Acoustic Doppler Current Profilers, Principles of operation: A Practical Primer, 1989 by RD Instruments, Inc.] RD Instruments suggests that the upper 15% of the range bins from a bottom-mounted ADCP should be discarded as unreliable, although this is just a simple rule of thumb for a rather complicated phenomenon.

With this understanding of the instrument as background it was easy for me to see what was wrong with the reported measurement. I pointed out the problem to the young scientist but he refused to listen. He excitedly explained to me that the instrument reported a quality value with each measurement, and that these near-surface measurements were of high "quality" and thus could not be wrong. The quality measurement that he referred to is a measurement of signal strength and so would not indicate a problem due to near-surface sidelobe contamination, which is a strong signal. I argued with him for a few minutes but I could not dissuade him. Unfortunately this type of unshakable faith in data is far too common an occurrence.

I explained the problem to the senior scientist on the experiment, and he understood immediately. Several days later, for my own amusement, I considered the ramifications of my young colleagues interpretation of the near-surface ADCP data. The Sound of Sleat is relatively open to the south, but connects again to the sea to the north through Kyle Rhea, a narrow channel of water between the Scottish mainland and the Isle of Skye. If the reported surface currents were accurate and if they extended over the entire mouth of the Sound then it was easy to compute the flows required through Kyle Rhea to achieve mass balance. The answer was over 30 m/s which I took to be just a tad high!

A good scientist not only understands their instruments but also questions every piece of data returned by those instruments. A typical trip to the movies or the theater involves a willing suspension of disbelief to enjoy the show. When considering data from field instruments you want to do just the opposite: you want to accentuate your disbelief. Only when you have been convinced that nothing is wrong with the instruments, or the way they are being operated, should you truly believe your data.

Redundancy

Always carry spare parts, spare instruments and spare computers. In fact, if you can, just take a spare for everything. On any given experiment it is not of question of if something will fail, it is a question of what will fail and when.

Here is what Sir Wyville Thomson of the Challenger Expedition in 1873 had to say on the subject (as quoted by Menard):

It is almost inconceivable how difficult it is to keep instruments, particularly those which are necessarily made of steel, in working order on board a ship; or how rapidly even with the greatest care they become destroyed or lost. For this reason it is necessary to have an almost unlimited supply of those in most frequent use, such as scissors, forceps, and scalpels of all sizes.

On a recent program I thought one of my acoustic current meters would make a great addition to a towed catamaran system that had been developed by some colleagues at Woods Hole. The designer of the catamaran wasn't easily convinced. We had never worked together before and I'm certain that he was a bit afraid that I would try to take credit for his new system. To convince him that I wasn't a threat and that the current meter would make a nice addition to his platform, I sat down and wrote out all of the reasons that the current meter should be added. I then wrote down my proposal for an agreement about how the data would be shared from the systems on the catamaran (see the section on collaborations).

My presentation won my colleague over and he agreed to include the current meter on his catamaran. I thought it strange, though, that he argued strenuously with one of my rationales for including the current meter, and that was redundancy. The measurements on the catamaran depended critically on knowing the orientation of the entire system. I argued that the current meter's built-in compass and tilt sensors could act as a nice backup to the catamaran's primary motion and orientation sensors. My colleague explained to me that my tilt sensors were not accurate enough (he was right) and that he had already built in two compasses. Still, my other arguments carried the day and the current meter was integrated into the system.

After a couple of days at sea, we began to realize that the main compass on the catamaran had failed. Whether it was due to damage in handling or some electrical problem I couldn't say. There were no spares for the primary compass, but we still had the secondary compass. A few more days passed and we started noticing that the secondary compass was drifting in some strange way. I discussed the problem with my colleagues. I was familiar with the compass they were using (see the previous section on knowing your instruments) so when they explained to me how the system was calibrated I knew what the problem was. They were using a small flux-gate compass with integral microcontroller. The microcontroller supported an automatic calibration mode that would recalibrate the compass when it passed through a 360° turn. I had used one of these compasses before and had spent some time figuring out how this auto-calibrate mode actually worked. My colleague at WHOI had read the manual and figured that if calibration was a good thing, then constant recalibration must be even better. He had locked the compass into a constant recalibration mode, so each time the catamaran would make a full circle, the compass would recalibrate itself!

Unfortunately, there was no way to undo the setting to the secondary compass in the field. Fortunately though, we had a backup to the backup compass inside my current meter. That compass essentially saved the entire scientific content of the catamaran system.

Of course redundancy should not be restricted to primary measurement systems. I always take a few general-purpose items along with me just in case they come in handy. For example, I always take one or two adjustable laboratory power supplies along with me. Power supplies are a common failure point in modern electronic systems. I can't provide a spare for every system, but a lab supply can be used to replace a faulty power supply in a pinch. In 1989 I was providing ground-truth measurements for a radar remote sensing experiment in a Scottish loch. Our equipment was all working, but I had heard that one of the other organizations had a problem with their radar. When I asked if there was something we could do to help out, I was told no, not unless I happened to have a spare ±15 volt power supply lying around. Well it just so happened that I did and the radar was fixed without a costly delay of several days in finding a replacement. The owners of the radar were amazed, but to me it is just common sense to take along not only everything that you need, but also everything that you might need.

I know that this may seem excessive, but I pride myself in taking along enough spare items that I usually end up helping out someone else's experiment as well as my own. The value of the reputation that you can develop by coming to the aid of your colleagues on an experiment cannot be underestimated.

I'll end this section with one final embarrassing story about the only time that I came back from an experiment totally empty handed. It was the second worst experience of my professional life. (The first was a disastrous presentation to a sponsor early in my career that to this day I don't want to talk about.)

Some colleagues at another laboratory built a large current meter array for measuring near-surface shears. Prior to them building this array I had pointed out that the S4 current meters that they had chosen for the array would not work because of magnetic crosstalk. S4s are electromagnetic current meters. They work by creating a magnetic field about themselves and then sense the voltage induced by the motion of the conducting sea water through this field. In order to separate out the S4's magnetic field from that of the Earth, the designers made the S4 field alternate polarity at a fixed frequency. When a pair of S4s are separated horizontally, they detect each other's fields and the resultant cross-talk can swamp out the current-induced signal. I had personally made this mistake before and knew they would have problems. My colleagues chose to ignore my warning. They had been told by the factory that this would not be a problem and they believed the factory instead of me!

[While this is an aside, there is a good lesson to be learned here. Always be suspicious of salesman. One of my colleagues at APL talks of catalog engineers. By this he means those people who believe what they read in the catalogs, design their equipment based on the published specifications, and then are surprised when things don't work as they planned. While most businesses are quite honest, the goal of all businesses is to sell product and make money. It is rare to find companies that will disclose the limitations of their own equipment, but there are a few. When faced with the choice of believing a scientist who claims a problem exists or believing a company's denial, I tend to believe the scientist. Many corporate claims can be easily dismissed if you understand enough about how their instrument works and ask enough questions. A company that answers all of your questions honestly and admits to the shortcomings of their instrumentation is a good one with which to do business. And frankly I give the same advice for buying stereos as for oceanographic equipment.]

It turned out that the crosstalk of the S4s on this array was so severe during the first deployment that the data were essentially useless. My colleagues then took the problem back to the factory, which modified the instruments to work on different frequencies to minimize the noise. At a presentation after the first experiment, but before the second experiment with the modified S4s, I again expressed reservations about the system. This time I was willing to put my money where my mouth was and offered to come along and add three of my ultrasonic current meters to their array to allow for a side-by-side comparison. To my surprise, they agreed to my proposal.

The experiment was a small engineering test, although given the size of the array and the number of people required to feed and tend it, the costs of the total test must have run at least $300K. My part of the test was a small add on which I estimated cost the sponsor about $30K, total. The relative costs become important later on as we shall see.

When I arrived, the array had already been assembled on the dock and the S4s were being installed. My colleagues had manufactured some mounting brackets for my instruments so it only took me a few hours to install my instruments. Overall I installed three current meters and the control and telemetry system from the APL wave spar onto their array. The control and telemetry was a bit over-engineered for this application, but it was already built and could easily handle the load of just these three instruments. Unfortunately, I made a terrible mistake in installing my system. When I assembled the control system, the rubber seal slipped as I was closing the cover, leaving a small gap through which water could enter. On the dock the defect went undiscovered and everything checked out fine.

The same could not be said of my colleague's system. They had ten S4 current meters multiplexed together into a single data stream by a control system that had been built specifically for this purpose by InterOcean, the manufacturer of the S4. The single data stream then fed into an RF modem for transmission back to the nearby ship. After some initial testing showed the system was ready to go, the array was hoisted into the water. The next day, just before beginning our tow, a set of newly charged batteries were swapped into the system on the array. Unfortunately, something then went wrong, and the system went dead. The exact reason for the failure was never discovered, but in the end the entire telemetry system had to be lifted back onto the dock and the deployment delayed. During this period my system continued to check out fine, so I pestered the technician working on the array telemetry problem with offers of help. After a full day of troubleshooting with no progress to show, my offers of help finally rubbed him the wrong way and we argued. I told him that when he finally gave up, I'd fix the system, but not a minute sooner. He went back to his work and I went off to have a beer.

Two more days passed, leaving just five days out of a planned eight-day test. Finally, I heard that the technician had diagnosed the problem as a bad telemetry transmitter. Unfortunately, he had no schematics for the unit (the manufacturer contended that the design was proprietary!) and he had no spare units or parts. He gave up. I went to the shop, asked him if I could fix it by replacing the entire telemetry system with a spare that I had. Given nothing to lose, he agreed.

The only problem I encountered in replacing their system was that my telemetry system required RS232 signal levels and the control computer had only a TTL-level interface. Fortunately I had some spare level-shifting ICs in my kit. So I wired up a small interface board in the shop. It looked like hell but it worked. I then put duct tape all around to insulate the board from its surroundings, taped it to the top of my spare telemetry system and taped the whole mess to the inside of their box. It was an ugly rats nest of wiring and duct tape, but it worked! From beginning to end I had taken about three hours to get their system up and working.

My plan for sharing the system was simple. I had three telemetry systems. One was in my telemetry box, one was now in my colleague's box on the array, and one was on board the ship. The plan was to set up my system at the beginning of the day's measurements to store my data internally. I would then switch off my telemetry system, switch on their system, and they could record their data in real time. Then at the end of the day we'd turn off their system and power mine back up so I could dump my data.

Unfortunately, when we arrived on station, my telemetry had taken on a few drops of water (its amazing how poorly electronics and sea water mix), and it no longer functioned. After some work and few spare board swaps I did manage to get everything in the system working but the data memory board. I had a backup for this board but I was missing one critical IC that I had robbed from the board months before in the lab for another project and had forgotten to replace. Thus I could get the system to communicate in real time, but I couldn't get it to store any data. There was thus no way that my colleague's system and my system could both work. To this day I am angry at myself for getting into this situation, but I made the only decision I could. I decided that they could use the telemetry system and my system would stay off. To my colleagues' credit they offered to share the channel with me, alternating between their instruments and my instruments, but this wasn't right. It was their experiment and their system was primary. I was supposed to be a low cost, non-interference add-on that was justified in terms of a comparison with their instruments. I made the decision that they should make full use of the telemetry channel. Though I came back empty handed, I'd make the same decision again.

The lessons from this story are many. If I had been a little more careful with the assembly of my system, it would not have leaked. If I had a full spare memory board with me then I could have repaired the system in the field. Also if I had a fourth telemetry package with me (I actually owned a fourth system, but it was broken at the time) then I could have set up two parallel channels. In the end I had enough spares to fix one problem but not both. Everyone I have subsequently told this story to have told me that I did the right thing, but in the end I came back empty handed. I was the one who suffered from the defeat. Just one more spare would have saved me this embarrassment. Take heed, because it can happen to you.

Instrumentation Checklist

• Gather the opinions and aid of experts when designing instruments, buoys, moorings or just about anything else when going to sea
• Expect things to fail, so examine critical components and single point failure modes
• Prepare for deployment using an instrumentation preparation checklist
• Completely test, characterize and calibrate every piece of equipment
  • Check power draw and thermal sensitivity against design specs
  • Assemble and test complete systems
  • Test for periods comparable to deployment, if at all possible
  • Check all batteries and replace if questionable
  • Keep all parts together and labeled when packing
  • Pack all require documentation
• Always pack spares of everything