Weird problem: devices working - not working - working - ...


#1

Hi fellow DMF enthusiasts,

We have had a DropBot system for two years and it has worked mostly fine all the time. During the past two weeks we have had a severe and weird problem. The problem is that sometimes the systems works, sometimes it doesn’t.

In practice what we have is that either water droplets are moving nicely at 100-120 V or they are not moving at all even at 160 V. We have been checking the voltage both in Microdrop and on device (on large inlet electrodes) with an oscilloscope, and whether the device works or not, the voltage is correct. The connection of the feedback clip has been secured all the time.

We have been totally puzzled, but finally today I remembered that we have the channel test PCB and we tried that. Attached pictures (1-3) show what we got from the three connectors. When this was done, the chip was not working. The results made me think the 0-39 switching board might be faulty, and since we have extra ones, we changed it. Then the channel test results were OK (pictures 4-6) and the device started working. After some time, when device was still working, we got again weird but different results from channel testing (pic 7).

This looks like board 0-39 might be broken, but it doesn’t make me sure the case is that simple. As I said, the chip have worked from time to time, and then again not worked. Just this morning I had a chip working, and after two hours, the same device was not working. Today, after changing the 0-39 board, the devices were working, but we didn’t have time to test for long time.

Is it possible that some part in the system is unstable and as a result, the feedback doesn’t sometimes work? What else could cause this behavior? Can the 0-39 board malfunctioning cause the other two boards to do the same? The problem is always on the whole device, not only parts of it.

Best regards,

Markus

···

Postdoctoral researcher

University of Helsinki

Finland


#2

Hi Markus,

This does sound like a strange problem; I can’t say I’ve ever experienced anything quite like what you are describing. In my experience, things have always been either broken or not. Debugging intermittent problems is definitely going to be more of a challenge. It also sounds like you are doing all of the right things to try and diagnose it. As long as the electrodes are getting the proper voltage (sounds like you’ve confirmed this with the oscilloscope), I can’t imagine why it shouldn’t work.

Just looking at your switching board test results, it looks like in (1), the current is much higher than expected for all channels. I would think this is probably some kind of short or damaged switch, but again, I wouldn’t expect this type of problem to be intermittent. (7) looks like the current is lower than expected for all switches (this can happen, for example, if the test board is not connected or if it is connected to the set of channels).

One thing I would suggest is testing the switching boards one at a time (i.e., disconnect all other switching boards from the system). Note that you will have to set their i2c address so that they show up as channels 0-39. If one of the boards is has a weird short, it could potentially influence the readings from the other boards.

Once you’ve got your system setup with a single HV switching board, try running the switching board test as you’ve been doing. If these results look reasonable (all values between ~0.5-1.5), try running the “device load” calibration procedure. The switching board test is done at a low voltage (~10V I think?), so the results are often noisy and we may be missing something subtle. The “device load” calibration is a bit more sensitive and easier to interpret. Please post the results from your “device load” calibration. If these look ok for at least one of your switching boards, we can “hopefully” rule out any problems with your control board’s feedback circuit, amplifier, etc.

-Ryan

···

On Wed, Sep 14, 2016 at 12:47 PM, Markus Haapala markus...@gmail.com wrote:

Hi fellow DMF enthusiasts,

We have had a DropBot system for two years and it has worked mostly fine all the time. During the past two weeks we have had a severe and weird problem. The problem is that sometimes the systems works, sometimes it doesn’t.

In practice what we have is that either water droplets are moving nicely at 100-120 V or they are not moving at all even at 160 V. We have been checking the voltage both in Microdrop and on device (on large inlet electrodes) with an oscilloscope, and whether the device works or not, the voltage is correct. The connection of the feedback clip has been secured all the time.

We have been totally puzzled, but finally today I remembered that we have the channel test PCB and we tried that. Attached pictures (1-3) show what we got from the three connectors. When this was done, the chip was not working. The results made me think the 0-39 switching board might be faulty, and since we have extra ones, we changed it. Then the channel test results were OK (pictures 4-6) and the device started working. After some time, when device was still working, we got again weird but different results from channel testing (pic 7).

This looks like board 0-39 might be broken, but it doesn’t make me sure the case is that simple. As I said, the chip have worked from time to time, and then again not worked. Just this morning I had a chip working, and after two hours, the same device was not working. Today, after changing the 0-39 board, the devices were working, but we didn’t have time to test for long time.

Is it possible that some part in the system is unstable and as a result, the feedback doesn’t sometimes work? What else could cause this behavior? Can the 0-39 board malfunctioning cause the other two boards to do the same? The problem is always on the whole device, not only parts of it.

Best regards,

Markus


Postdoctoral researcher

University of Helsinki

Finland

You received this message because you are subscribed to the Google Groups “dropbot-dev” group.

To unsubscribe from this group and stop receiving emails from it, send an email to dropbot-dev+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.


#3

Hi Ryan,

We just tested the switching

boards separately. We tested all the “old” ones and one new board. There is obviously a problem with the old 0-39 board but I’m not sure what the results should look like and if the other three boards are OK. Pictures 1-4 have the channel testing results and pictures 5-9 have the device load calibration.

We will now put the system back together with the 0-39 board replaced with a new one.

Br,

Markus

···

2016-09-15 16:06 GMT+03:00 Ryan Fobel ry...@fobel.net:

Hi Markus,

This does sound like a strange problem; I can’t say I’ve ever experienced anything quite like what you are describing. In my experience, things have always been either broken or not. Debugging intermittent problems is definitely going to be more of a challenge. It also sounds like you are doing all of the right things to try and diagnose it. As long as the electrodes are getting the proper voltage (sounds like you’ve confirmed this with the oscilloscope), I can’t imagine why it shouldn’t work.

Just looking at your switching board test results, it looks like in (1), the current is much higher than expected for all channels. I would think this is probably some kind of short or damaged switch, but again, I wouldn’t expect this type of problem to be intermittent. (7) looks like the current is lower than expected for all switches (this can happen, for example, if the test board is not connected or if it is connected to the set of channels).

One thing I would suggest is testing the switching boards one at a time (i.e., disconnect all other switching boards from the system). Note that you will have to set their i2c address so that they show up as channels 0-39. If one of the boards is has a weird short, it could potentially influence the readings from the other boards.

Once you’ve got your system setup with a single HV switching board, try running the switching board test as you’ve been doing. If these results look reasonable (all values between ~0.5-1.5), try running the “device load” calibration procedure. The switching board test is done at a low voltage (~10V I think?), so the results are often noisy and we may be missing something subtle. The “device load” calibration is a bit more sensitive and easier to interpret. Please post the results from your “device load” calibration. If these look ok for at least one of your switching boards, we can “hopefully” rule out any problems with your control board’s feedback circuit, amplifier, etc.

-Ryan

You received this message because you are subscribed to the Google Groups “dropbot-dev” group.

To unsubscribe from this group and stop receiving emails from it, send an email to dropbot-dev+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

On Wed, Sep 14, 2016 at 12:47 PM, Markus Haapala markus...@gmail.com wrote:

Hi fellow DMF enthusiasts,

We have had a DropBot system for two years and it has worked mostly fine all the time. During the past two weeks we have had a severe and weird problem. The problem is that sometimes the systems works, sometimes it doesn’t.

In practice what we have is that either water droplets are moving nicely at 100-120 V or they are not moving at all even at 160 V. We have been checking the voltage both in Microdrop and on device (on large inlet electrodes) with an oscilloscope, and whether the device works or not, the voltage is correct. The connection of the feedback clip has been secured all the time.

We have been totally puzzled, but finally today I remembered that we have the channel test PCB and we tried that. Attached pictures (1-3) show what we got from the three connectors. When this was done, the chip was not working. The results made me think the 0-39 switching board might be faulty, and since we have extra ones, we changed it. Then the channel test results were OK (pictures 4-6) and the device started working. After some time, when device was still working, we got again weird but different results from channel testing (pic 7).

This looks like board 0-39 might be broken, but it doesn’t make me sure the case is that simple. As I said, the chip have worked from time to time, and then again not worked. Just this morning I had a chip working, and after two hours, the same device was not working. Today, after changing the 0-39 board, the devices were working, but we didn’t have time to test for long time.

Is it possible that some part in the system is unstable and as a result, the feedback doesn’t sometimes work? What else could cause this behavior? Can the 0-39 board malfunctioning cause the other two boards to do the same? The problem is always on the whole device, not only parts of it.

Best regards,

Markus


Postdoctoral researcher

University of Helsinki

Finland

You received this message because you are subscribed to the Google Groups “dropbot-dev” group.

To unsubscribe from this group and stop receiving emails from it, send an email to dropbot-dev+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.


#4

Hi again,

Now the system works nicely with a new 0-39 switching board.

I forgot to mention one thing in the second email and I think it makes things even more interesting. The voltage had a different waveform when we had the intermittent problem of device working and not working. Today, after replacing the board, we had a nice sine wave but before it was a more squarish kind of wave. The voltage was correct and the waveform was exactly the same regardless of if the device was working or not.

Does this point to a different point of failure than the switching board? I don’t know much about the design but it seems weird that the switching board would change the waveform generated upstream in the signal generator.

···

Br,

Markus

2016-09-15 19:11 GMT+03:00 Markus Haapala markus...@gmail.com:

Hi Ryan,

We just tested the switching

boards separately. We tested all the “old” ones and one new board. There is obviously a problem with the old 0-39 board but I’m not sure what the results should look like and if the other three boards are OK. Pictures 1-4 have the channel testing results and pictures 5-9 have the device load calibration.

We will now put the system back together with the 0-39 board replaced with a new one.

Br,

Markus

2016-09-15 16:06 GMT+03:00 Ryan Fobel ry...@fobel.net:

Hi Markus,

This does sound like a strange problem; I can’t say I’ve ever experienced anything quite like what you are describing. In my experience, things have always been either broken or not. Debugging intermittent problems is definitely going to be more of a challenge. It also sounds like you are doing all of the right things to try and diagnose it. As long as the electrodes are getting the proper voltage (sounds like you’ve confirmed this with the oscilloscope), I can’t imagine why it shouldn’t work.

Just looking at your switching board test results, it looks like in (1), the current is much higher than expected for all channels. I would think this is probably some kind of short or damaged switch, but again, I wouldn’t expect this type of problem to be intermittent. (7) looks like the current is lower than expected for all switches (this can happen, for example, if the test board is not connected or if it is connected to the set of channels).

One thing I would suggest is testing the switching boards one at a time (i.e., disconnect all other switching boards from the system). Note that you will have to set their i2c address so that they show up as channels 0-39. If one of the boards is has a weird short, it could potentially influence the readings from the other boards.

Once you’ve got your system setup with a single HV switching board, try running the switching board test as you’ve been doing. If these results look reasonable (all values between ~0.5-1.5), try running the “device load” calibration procedure. The switching board test is done at a low voltage (~10V I think?), so the results are often noisy and we may be missing something subtle. The “device load” calibration is a bit more sensitive and easier to interpret. Please post the results from your “device load” calibration. If these look ok for at least one of your switching boards, we can “hopefully” rule out any problems with your control board’s feedback circuit, amplifier, etc.

-Ryan

You received this message because you are subscribed to the Google Groups “dropbot-dev” group.

To unsubscribe from this group and stop receiving emails from it, send an email to dropbot-dev+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

On Wed, Sep 14, 2016 at 12:47 PM, Markus Haapala markus...@gmail.com wrote:

Hi fellow DMF enthusiasts,

We have had a DropBot system for two years and it has worked mostly fine all the time. During the past two weeks we have had a severe and weird problem. The problem is that sometimes the systems works, sometimes it doesn’t.

In practice what we have is that either water droplets are moving nicely at 100-120 V or they are not moving at all even at 160 V. We have been checking the voltage both in Microdrop and on device (on large inlet electrodes) with an oscilloscope, and whether the device works or not, the voltage is correct. The connection of the feedback clip has been secured all the time.

We have been totally puzzled, but finally today I remembered that we have the channel test PCB and we tried that. Attached pictures (1-3) show what we got from the three connectors. When this was done, the chip was not working. The results made me think the 0-39 switching board might be faulty, and since we have extra ones, we changed it. Then the channel test results were OK (pictures 4-6) and the device started working. After some time, when device was still working, we got again weird but different results from channel testing (pic 7).

This looks like board 0-39 might be broken, but it doesn’t make me sure the case is that simple. As I said, the chip have worked from time to time, and then again not worked. Just this morning I had a chip working, and after two hours, the same device was not working. Today, after changing the 0-39 board, the devices were working, but we didn’t have time to test for long time.

Is it possible that some part in the system is unstable and as a result, the feedback doesn’t sometimes work? What else could cause this behavior? Can the 0-39 board malfunctioning cause the other two boards to do the same? The problem is always on the whole device, not only parts of it.

Best regards,

Markus


Postdoctoral researcher

University of Helsinki

Finland

You received this message because you are subscribed to the Google Groups “dropbot-dev” group.

To unsubscribe from this group and stop receiving emails from it, send an email to dropbot-dev+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.


#5

Hi Markus,

Glad to hear you are back up and running! The test results from your previous email look consistent with what we’ve seen on our systems here (i.e., RMSE is <5% for the vast majority of frequency/load combinations).

Regarding the waveform shape, seeing a non-sine wave output is definitely a red flag that something is wrong. I suspect that your “broken” switching board a short, which may be over-loading the amplifier and producing this behavior. To test this theory, you can use an oscilloscope to compare the waveform shape of the low-voltage output of the signal generator with the high-voltage output coming from the amplifier.

If you want to try debugging and fixing that switching board, I would first check the resistance between the HV and HVGND channel (this should be >1MOhm). Make sure you have nothing hooked up to the gold SMA connectors on the board when you do this (but the board should be powered on). You may also want to cycle through each individual switch on the broken board using a multimeter (checking the conductivity between the switch output and both HV and HVGND) to try and locate the source of the problem. Most likely, there is a bad PhotoMOS on there that you could replace to get the board working again.

-Ryan

···

On Thu, Sep 15, 2016 at 3:26 PM, Markus Haapala markus...@gmail.com wrote:

Hi again,

Now the system works nicely with a new 0-39 switching board.

I forgot to mention one thing in the second email and I think it makes things even more interesting. The voltage had a different waveform when we had the intermittent problem of device working and not working. Today, after replacing the board, we had a nice sine wave but before it was a more squarish kind of wave. The voltage was correct and the waveform was exactly the same regardless of if the device was working or not.

Does this point to a different point of failure than the switching board? I don’t know much about the design but it seems weird that the switching board would change the waveform generated upstream in the signal generator.

You received this message because you are subscribed to the Google Groups “dropbot-dev” group.

To unsubscribe from this group and stop receiving emails from it, send an email to dropbot-dev+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Br,

Markus

2016-09-15 19:11 GMT+03:00 Markus Haapala markus...@gmail.com:

Hi Ryan,

We just tested the switching

boards separately. We tested all the “old” ones and one new board. There is obviously a problem with the old 0-39 board but I’m not sure what the results should look like and if the other three boards are OK. Pictures 1-4 have the channel testing results and pictures 5-9 have the device load calibration.

We will now put the system back together with the 0-39 board replaced with a new one.

Br,

Markus

2016-09-15 16:06 GMT+03:00 Ryan Fobel ry...@fobel.net:

Hi Markus,

This does sound like a strange problem; I can’t say I’ve ever experienced anything quite like what you are describing. In my experience, things have always been either broken or not. Debugging intermittent problems is definitely going to be more of a challenge. It also sounds like you are doing all of the right things to try and diagnose it. As long as the electrodes are getting the proper voltage (sounds like you’ve confirmed this with the oscilloscope), I can’t imagine why it shouldn’t work.

Just looking at your switching board test results, it looks like in (1), the current is much higher than expected for all channels. I would think this is probably some kind of short or damaged switch, but again, I wouldn’t expect this type of problem to be intermittent. (7) looks like the current is lower than expected for all switches (this can happen, for example, if the test board is not connected or if it is connected to the set of channels).

One thing I would suggest is testing the switching boards one at a time (i.e., disconnect all other switching boards from the system). Note that you will have to set their i2c address so that they show up as channels 0-39. If one of the boards is has a weird short, it could potentially influence the readings from the other boards.

Once you’ve got your system setup with a single HV switching board, try running the switching board test as you’ve been doing. If these results look reasonable (all values between ~0.5-1.5), try running the “device load” calibration procedure. The switching board test is done at a low voltage (~10V I think?), so the results are often noisy and we may be missing something subtle. The “device load” calibration is a bit more sensitive and easier to interpret. Please post the results from your “device load” calibration. If these look ok for at least one of your switching boards, we can “hopefully” rule out any problems with your control board’s feedback circuit, amplifier, etc.

-Ryan

You received this message because you are subscribed to the Google Groups “dropbot-dev” group.

To unsubscribe from this group and stop receiving emails from it, send an email to dropbot-dev+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

On Wed, Sep 14, 2016 at 12:47 PM, Markus Haapala markus...@gmail.com wrote:

Hi fellow DMF enthusiasts,

We have had a DropBot system for two years and it has worked mostly fine all the time. During the past two weeks we have had a severe and weird problem. The problem is that sometimes the systems works, sometimes it doesn’t.

In practice what we have is that either water droplets are moving nicely at 100-120 V or they are not moving at all even at 160 V. We have been checking the voltage both in Microdrop and on device (on large inlet electrodes) with an oscilloscope, and whether the device works or not, the voltage is correct. The connection of the feedback clip has been secured all the time.

We have been totally puzzled, but finally today I remembered that we have the channel test PCB and we tried that. Attached pictures (1-3) show what we got from the three connectors. When this was done, the chip was not working. The results made me think the 0-39 switching board might be faulty, and since we have extra ones, we changed it. Then the channel test results were OK (pictures 4-6) and the device started working. After some time, when device was still working, we got again weird but different results from channel testing (pic 7).

This looks like board 0-39 might be broken, but it doesn’t make me sure the case is that simple. As I said, the chip have worked from time to time, and then again not worked. Just this morning I had a chip working, and after two hours, the same device was not working. Today, after changing the 0-39 board, the devices were working, but we didn’t have time to test for long time.

Is it possible that some part in the system is unstable and as a result, the feedback doesn’t sometimes work? What else could cause this behavior? Can the 0-39 board malfunctioning cause the other two boards to do the same? The problem is always on the whole device, not only parts of it.

Best regards,

Markus


Postdoctoral researcher

University of Helsinki

Finland

You received this message because you are subscribed to the Google Groups “dropbot-dev” group.

To unsubscribe from this group and stop receiving emails from it, send an email to dropbot-dev+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.