10*N Rocket Launch ...
 
Notifications
Clear all

10*N Rocket Launch Control System

33 Posts
3 Users
9 Likes
3,605 Views
(@davee)
Member
Joined: 3 years ago
Posts: 1606
 

Hi @lydara et al,

 I have actually been trying to ponder the task from a slightly more philosophical/requirements approach. Traditional systems design is all about top-down, doing requirements first, then moving towards implementation, and whilst I am not convinced it works if it is done in a puristic way, I think it has merit if you allow pragmatic implementation thoughts when they naturally appear.  There is a danger of this going 'over the top', but maybe humour me for now, as I hope some good may come of it.

I should like to start by 'setting the scene' as I see - with the aim of establishing whether your 'on the ground' experience and thoughts, match my eclectic armchair imagination.

For brevity, (not arrogance), I have written in the style of an informal but 'authoritative' specification, but it is really only a suggestion. Please feel free to correct where I have got it wrong!!

---------------

Looking quickly at your circuits, I think you were heading in a similar direction at both the power source (battery) end and towards the igniters, albeit I might propose using multi-contact relays for the latter. The main differences seem to be (1) trying to formally separate some parts and (2) adding some 'ROS' remote control to the igniter feeds at a 'group' level, rather than relying on a manual box switch. My aim is to keep the 'high integrity' requirements in the realm of simple things like switches and relays, and allow more complex functions like microcontrollers to play their part in Low Integrity areas, where a failure will be an annoyance, but not a hazard. I note you have some additional functions like strobes and displays - For the moment I have not considered them, but hope they can be tagged on to the Low Integrity parts later, as required.

----------

I'll define/adopt box names:

  1. 'Pad Box' :  remotely controlled box, nearer to the rockets.
  2. 'ROS Box' : Master user (remote) control centre
  3. 'Obs Box'  : Simple remote control for 'Scout centred' observer/supervisor .. minimally 1 switch that can abort firing procedure, in series with a removeable plug/token
  4. 'Launch Box' : Simple remote control for each scout .. minimally 1 button

-----------

By the way, I shall breaking the system into 'bits' during this discussion .. these 'bits' may not fit neatly into physical boxes  .. try not to get too concerned about that for now... hopefully it will become clearer.

And I am using 'italics' to suggest 'discussion', 'normal font' for 'specification'.

----------

The 'system design' should include specification of the personnel and their roles, charged with controlling usage of the equipment etc. I would envisage a minimum of one person (ROS?) responsible for the overall management and control of the 'master' control unit. This person shoud be fully trained/competent/conversant with the entire operation and system. They should not be responsible for 'managing' scouts during the firing preparation and count down operations, as their full attention should be directed to the safe overall operation of the equipment and site.

In addition, there should be (at least) one other responsible person for each 'group' of scouts (up to 10 say?), whose duties would include watching the firing area during the firing sequence. (it is implied this grouping will match the equipment grouping.) This other person could be provided with a simple control unit that can immediately disable the firing mechanism if they see a hazard, but not enable it.  Optionally, this control unit could also require a token (plug say) to be inserted, to indicate the person is 'on station'; inserting the plug would not enable the firing mechanism, but it could be mandatory for the token(s) to be in place before the firing countdown can be initiated, and immediately aborted if (any) token is removed. Whilst this person should be fully aware and 'comfortable' with the entire day's operation, and to be fully trusted/responsible, they would only need an overall appreciation of the technical details of the system and its controls, etc., mainly regarding procedures if an emergency arises.

----------

The system design/implementation task, even in its most basic form, includes considerable amounts of wires, switches, and so on, spread over many metres of space, which creates a natural complexity, cost, etc., and all of the equipment has to built, maintained, checked for faults, knowing that some faults, if not corrected in a timely manner, could have fatal consequences. To compound that, the majority of 'users' of the system are excited, untrained (with respect to technical system), young scouts, some of whom are likely to try to 'game' the system. Hence, you need a system which has a clear safety approach that is not buried in the complexity of hundreds of switches, plugs and so on.

I suggest we try to break the system into three integrity categories:

  1. High Integrity
    • This part of the system should ("alone" - in spite of how the Medium and Low integrity parts behave) make it impossible to ignite a rocket at any time other than when this part of the system is explicitly enabled by the ROS (and responsible adult helper's contribution).
    • It should be designed so that failure of any individual component should not result in a hazardous/tragic situation, (such as igniting a rocket) nor should it compromise the previous requirement of preventing a rocket being ignited at the wrong time. It should be simple in concept, so that its design can be assessed, etc.
    • Malicious/unauthorised changes are likely to cause hazards and possibly tragedy.
    • Equipment seals or other measures to prevent/warn of interference should be considered.
    • It must be in the physical control (partly of 'ROS' or responsible adult helper/supervisor at all times that misuse could result in a hazard.
  2. Medium Integrity:
    • It is not directly expected to play a significant role in preventing unauthorised or unexpected rocket ignition.
    • Unless subjected to gross damage or deliberate physical attack (such as maliciously changing the internal wiring), it should not be capable igniting a rocket.
    • Equipment seals or other measures to prevent/warn of interference should be considered.
    • It will be in the physical control of 'ROS' or responsible adult helper/supervisor at all times that misuse could result in a hazard.
  3. Low Integrity - This part of system shall not be capable of igniting a rocket (or any other hazard to be identified later) in spite of it possibly containing any number of undiscovered faults due to poor design, be subject to misuse and abuse, including malicious rewiring, and be suffering multiple component failures.

-----------------------

Assuming the above personnel scenario and equipment integrity requirements are 'approved', then the next 'trick' is to propose an implementation. Clearly a full implementation design will take a while, so for now I just intend to give some 'clues' and invite feedback as to whether is worth pursuing.

------------------------

Power source - Medium Integrity - Part of 'Pad Box'

  1. This is in a remotely controlled unit, relatively close to the rockets,
  2. This is a 12V rechargeable battery, with master manual battery disconnect switch, fuse, indicator, charging socket/provision.
  3. Two (maybe three) battery outputs (all controlled by battery disconnect switch):
    • Control power - via fuse, with indicator - powers controls
    • Igniter power - via switch and fuse, with indicator - Note switch is for 'extra safe mode testing' - expected to be closed most of the time - omitting it should not affect safety analysis
    • Igniter Relay Coil Power - there is case for a third feed for the igniter relay coils, as a 'dirty power' adjunct for the Control Power
    • (A case might also be made to power the 'system relays' as second 'dirty source' ... clearly this power would need to be switched on for most functionality .. I will leave considering it to a later design phase.)

These 'power outputs' are internal to the physical 'Pad Box' unit, and pass to the 'Power Control' defined next .. they are conceptual outputs, when analysing the safety case.

Power Control - High Integrity

This is the section that determines the safety case. Deal with each Power Source output from the previous section above in turn:

  1. Control power
    • This might be reduced to Medium Integrity, depending on detail design decisions
    • this powers all of the microcontrollers/sensors/indicators etc as required in the various boxes.
    • It is expected to consist of a small number of separate wires, each with overcurrent protection.
    • Presently it is expected these outputs will be 'always on', and not part of any safety case for launch control.
  2. Igniter Power
    • This is the 'centre' of the High Integrity - every effort must be made to ensure the relay switches in this section only allow this power to flow when it is safe
    • This only supplies power to the igniters (plus any direct power status indicator lights, etc.)
    • For this section to supply power, all of the following must be simultaneouly asserted:
      • ROS controller has manually selected 'launch enable'
      • A timer which provides appropriate delays and launch time window 'approves launch'
      • All of the Observers/scout supervisors' are 'on station' and holding 'launch enable'
      • All other safety systems are saying 'All clear to launch'
    • In principle, each of the systems above could have an independent relay in series, such that they form an 'AND' gate. In practice, a case could be made for reducing the number of relays, using appropriate electronics, etc. to combine the asserted approvals, but it certainly should not be less than two relays and the electronics must be very simple. A trick that may be appropriate is to independently control both the power feed and the power return to each coil. e.g. Observer's OK supplies +12V, All other systems provide the 0V return.
    • In general, these relays must be controlled by 'simple', manual switches and plugs, so complex hardware is not part of the safety case. An exception for the timer function could be proposed if you can show that it is more for 'convenience' than 'safety'.
    • When a candidate arrangement is proposed, it must carefully checked, considering what might happen if any component in this section, or the asserted signal, becomes jammed or corrupted. THIS IS THE HEART OF THE SAFETY CASE!
    • Some thought should be given as to whether both connections to the igniter, either individually, or as a group should be switched. You may wish to consult if you are not already familiar with the answer to that question.
    • Relays in this section shall continually send their open/closed status for both ROS status display and microcontroller comparison. (Also maybe display on the PadBox itself, but not to rely on that in 'live usage'.)
  3. Igniter Relay Coil Power
    1. This could be a simple feed which is 'always on' when Control Power is on. It would help to reduce the risk of transients upsetting the other electronics.
    2. Alternately, it could become part of the 'safety case' if one or more of the protection systems prevented the relay coils being energised. This might have the disadvantage of it being more difficult to 'dry run' test the system, but this might not be a deal breaker.

Rest of PAD Box - Medium Integrity

The other main function of the PAD box consists of demultiplexing the igniter fire commands, instigated by the scouts pressing their fire buttons, and closing the individual igniter relays. Because the igniter power sourced to the relay contacts is only available for a short period of time, carefully controlled by the subsystem described above, it is now conveniently reduced to a Low Integrity function. However, it is still necessary to take great care with the physical design and layout to ensure that a 'stray' power source cannot 'spoof' the system. And of course, any malicious 'rewiring' could be disasterous.

Hence, this appears to qualify as a Medium Integrity case, which should be manageable with care. It does not depend upon proving the demultiplexing and other functions are safe, which I think would be near impossible.

Controlling the coil power, might provide an extra level of assurance, and is worth considering when the detail design is undertaken.

-------------------------------------

ROS Box - Igniter Power Enable Functions - High Integrity

As discussed previously, the core of the safety assessment is showing that power is only supplied to individual igniter relays for the very brief window of time when all of the necessary checks of the equipment functionality and checking that the launchpad is clear of personnel has been achieved.

The required control function is a combination of switches/plugs controlled by ROS directly at the ROS Box, remote switch/plug lines from the OBS Box(es), and the timer function.

For this brief window of time, the ROS Box is responsible for making the necessary connections to activate the coils of the relays within the Pad Box. With the possible exception of the timer, it is expected this will be achieved by simple switches, plugs, etc.

Clearly this is a high integrity function, but hopefully is not too complex.

Some thought will be needed to guard against faulty cables, switches, etc. I presume this likely to be a manual or semi-manual task, mainly prior to an event. Perhaps the main case is if ROS manual enable path is compromised ... some thought may be needed to split it into two paths so that a single fault cannot bypass both paths.

ROS Box - Launch Box Multiplexer - Low Integrity

This may complex, with microcontrollers, etc, but it is now low integrity, so software faults, etc, should not need to be considered, because at most it can only close igniter relays with no power on their contacts.

Each LaunchBox could be just a simple switch .. the previous token requirement has been migrated to the OBS box(es). Whilst this implies one or more personnel, I should have thought they were essential, given the type of activity. And managing one token is surely easier than managing 10?

As elsewhere, gating the 'launch' commands in the ROS box with overall igniter enable decisions for extra confidence is optional. Care with design to ensure the subsystems have no direct connection to crossfeed.

ROS Box - OBS Box(es) - High Integrity

This is an additional personnel function which provides a highly optimised, thrifty, 'wet-ware' scanning system of the launch site, does not require a deep technical knowledge, but performs a useful abort/allow_to_continue monitoring and supervisory function .. the hardware is possibly just a plug/socket and biased-off switch in series. 

-------------------------------------------------------------------------------------

I don't know if this diatribe is comprehensible ... I hope so. Obviously a few diagrams would help but I wanted to get a story together and adding diagrams would take me an age.

Let me know what you think. Do you think this approach could meet your safety assessment requirements? Dave

 


   
LydaRA reacted
ReplyQuote
(@davee)
Member
Joined: 3 years ago
Posts: 1606
 

Hi @lydara,

  I have just seen your recent reply ... after posting my diatribe .. I don't think there are any contradictions ... providing we assume Launch Control Officer and Observer/Scout Supervisor are the same person(s). I had assumed the ROS would need to provide some form of key or token to enable the control, as well using any switches, etc. on the panel

Hope it helps. Dave


   
LydaRA reacted
ReplyQuote
(@lydara)
Member
Joined: 3 years ago
Posts: 90
Topic starter  

@davee  Thanks for so much thought.  Yes, I think you've got most of it.  Ultimately the Range Safety Officer is the one with knowledge & authority running the range.  The Scouts/Launch Control Officers are being given a sense of empowerment, to educate & inspire. 

And you are correct, no one guy could "herd this many cats."  There are so many rockets to wire & mount, so many children actively engaged on the rage, and more children waiting their turn.  Aside from how obvious that workload is, BSA Youth Protection rules will not let just one guy try to handle the children.  So, I usually have two to three Youth Staff (teenage volunteers) who help educate (children sometimes relate better to those closer to their own ages, & we want to give these teenagers opportunity for public speaking, leadership growth, and service).  And as these groups of children camp & move from station to station, they have to have their own "two-deep" set of multiple leaders (ones they know & who know how to motivate particular individuals).

In fact, as we are seeking to increase the rocketry station's throughput, I've even thought of minimal rocketry training for the traveling leaders.  We always ask them to chime in and "make it real" by sharing how they define & use "tag-out/lockout" in their own professions.  Now if we teach them, these adults could also mount engines & igniters, clip on the alligators, and mount to the launch rods.

I have an entire discussion of UI, packaging, design, & testing philosophy going with a lead official at the National Association for Rocketry.  This is the standards body that the Boy Scouts of America has delegated our safety rules to.  And while we used to just take the best-qualified volunteer to run the rocketry range we could get, after the slack operations of my predecessor, I convinced the local Council to require that we train such rocketry station leaders in NAR Safety Code and low power model rocketry at a minimum.  We also prefer a NRA shooting instructor or RSO and/or a professional engineer or electrician.  The station also seeks augmented Weather Hazards & First Aid/CPR training for all leaders & Youth Staff members.


   
MadMisha and DaveE reacted
ReplyQuote
(@davee)
Member
Joined: 3 years ago
Posts: 1606
 

Hi @lydara,

  You are obviously very busy and dedicated to the cause - it sounds like it is expanding into an industry in its right! You must be hero with the scouts over a wide area!!

And I am somewhat relieved to hear that you are not trying to do the live firing on your own ... I didn't think you could be, but there wasn't much evidence to go on.

I don't know if my explanation was clear enough, but I aimed to come up with the barest bones of a framework, which can in principle be checked analysed for safety, if properly detailed and documented. If that is successful, then obviously it can expanded to include more details to include microcontrollers and the like, of which you have already suggested some approaches. If you have any queries that I might be able to help you with, please let me know.

I am still not sure if a mechanised and thrifty system for automatically scanning the 'stage' is viable - obviously the self-drive car market, amongst others, are trying to resolve similar and harder questions, but not with your budget and resource constraints ... and reading the press, not with a 100% safety record either.

I am sure some simple methods could be tried, and they will probably detect correctly some of the time. It is true that even a '90%' correct detection system might save someone from harm .. but unfortunately if it causes false alarms or creates a false sense of security, it can be worse than no system. Conveniently, if you discover a reliable detection system, then I left a slot for it under 'All other safety systems'! 😀 

I hope that it of some use to you and you can move forward with your project. Dave


   
LydaRA reacted
ReplyQuote
(@lydara)
Member
Joined: 3 years ago
Posts: 90
Topic starter  

@davee  Thanks.  Aside from the time with my own son (who just _had_ to outgrow Cub Scouts, and is himself a Youth Staffer this year...), seeing this many children have fun and get excited about engineering...  Well these two weeks of rocketry is one of the highlights of my year!  I am just one of many volunteers it takes to put on a camp.  We should all do whatever we can to give the mothers a break and to inspire the next generation--they are worth it!

While I connect with the youth well, my weakness for this add-on project is the electronics to implement it.  30+ years ago, I learned basic electronics as a Red Cross volunteer at a hospital for the mentally retarded in Georgia.  My mentor taught me to help build motorized wheelchairs (the State could not afford enough of these commercially) and to build adaptations for individuals (big buttons for elbows, mercury-switch-thumb-extensions, tap-coded-interpretation controls, etc. to amplify whatever physical control they had).  But that was long ago.  I graduated from the magnet engineering high school, then Wofford College, and then went into _using_ and _supporting_ I.T. systems (well 1/3 programming)--but really not any more construction of hardware.  And getting back into this now....holy cow, have the number of component options increased exponentially!  No more Radio Shack to grab things from locally, but now whole new classes of components to learn & datasheets to read...  And with things being mail-ordered, no one locally to ask for suggestions!

So again, I apologize to all for the amateur-hour nature of my electrical schematics.  But hey, at least I know enough to stop and ask for more eyeballs on this before I mess it up completely!  :-}  Thanks again for all suggestions.


   
DaveE reacted
ReplyQuote
MadMisha
(@madmisha)
Member
Joined: 4 years ago
Posts: 340
 
Posted by: @lydara

So again, I apologize to all for the amateur-hour nature of my electrical schematics.

I have seen far worse. Correction, I have made far worse!


   
DaveE reacted
ReplyQuote
(@davee)
Member
Joined: 3 years ago
Posts: 1606
 

Hi @lydara,

  And what more can I add to a great story like that ... well done to you!!

I also remember Radio Shack/Tandy, although their influence in the UK was much smaller than in their native US, but of course there were other local suppliers - indeed I 'ran' the parts counter at the back of an independent Radio/TV shop for a couple of years as an after school/Saturday job some 50 years ago! Sadly, they have all pretty much gone by the wayside and we are left a few big multi-nationals who mainly seem to interested in buying each other, some semiconductor companies are reasonably friendly, and taking our chance with the Banggood and AliExpress bazaars. Sometimes, we can get some amazing things for the price of a coffee, other times junk or outrageous costs. Of course, the WWW/Internet is the biggest change.

-------------

And your schematics need no apologies. .. I encourage you to 'dust them off', not they are old enough to have gathered more than a few specs of dust, when you are ready.

Most 'DIY' projects you will see discussed are safe-ish, regardless of their design and implementation flaws, whilst your project not only requires the 'usual' skills, it also demands a very high safety ethic, and that is tough area to satisfy - it is about ensuring bad outcomes are always impossible - most projects are happy if they work on a 'good' day - safety-critical stuff has to play at least safely EVERY day.

You may well have already know the following, but as it is tricky for us to have a chat over a coffee in your kitchen, I hope you don't mind me trying to explain. My explanation will probably be a bit of muddle, so please query anything that looks unexpected, unclear or wrong.

I tried to take the essence of where you were heading, maybe tweak it a little bit, and describe what I hoped you could convert into a safe framework for the safety critical bits. I may or may not have succeeded, but I was trying to demonstrate a way forward that didn't require every aspect to be safety critical, since there was no way you could design the whole system with microcontrollers, etc., all to the highest safety-critical standard. And if it is 'all in together', it will be difficult to explain the safety-critical core to anyone else... which I assume you will need to do at some point.

So, somehow I suggest you split the 'must never fail' stuff from all of the 'nice to have/makes it convenient and smart' stuff, and hope you can design and build the 'must never fail' bit in such a way that it is 'obviously safe', and furthermore that part embues the whole system with the required, 'will never fail with catastrophic consequences'. Please don't think is easy, but I hope it will be possible.

I think you had a reasonable starting basis before, albeit mixed with the non-safety critical bits..

.. I suggested adding a small number of relays to

  • 'bring the safety critical power path together' in the Pad box
  • completely disable the firing mechanism remotely from the RSO box, so that not even the RSO has to approach the Pad Box to disable the firing system
  • allow the LCO to be part of the enable system
  • simplify the scout control (since it no longer part of the safety critical system)

Also the scheme resulted in relays in series, so that a single component failure, like a relay sticking closed, could not undermine the isolation.

-------

Anyhow, I recommend you try to extract the best of your ideas from your schematic DIRECTLY relating to this safety-critical core, change it as you think best, and produce a basic schematic for a 'minimal' system.

This minimal system, should not implement any complexities like multiplexers, networks, etc. or have any other adornments like strobes. I would leave the timer out ..  assume/show how the RSO can perform this role with his switches... if necessary, provide him an extra switch, labelled timer that he controls with his third hand!

I suggest it assumes 1 RSO, 1 LCO and 1 (over-excited, so he will try but fail to prematurely launch) scout, with direct wiring to relays etc., and 1 rocket.

---------

I am sure there are at least one or two people you have been in touch with to advise on any queries or cast a beady eye over your thoughts. (If you are unlucky, I will chip in!)

You then need to try to get this minimal architecture/framework approved or validated ... certainly to your own satisfaction, but preferably to that of the National Association of Rocketry. (Assuming that is the appropriate body.)

Once you have such a framework, then you can introduce the 'fancy' stuff with microcontrollers and so on, which will not of course be GUARANTEED to be faultfree - you just have to ensure the additions cannot break your framework in such a way that if affects its safety.

I would recommend adding sophistication is broken into stages or phases. You will of course need to take the NAR on that upgrade path, but hopefully that will be straightforward if you have gained their confidence with the safe 'core' system.

Hope this helps you to get a little further .. sorry brevity is not my middle name!

Best wishes, Dave


   
ReplyQuote
(@lydara)
Member
Joined: 3 years ago
Posts: 90
Topic starter  

@davee  Yet again, we think alike. 

The RSO's Master Arm removable key is the most critical part of the power plan.  Before he ever leaves the firing line headed for the launch pads, the RSO puts this key in his pocket.  As soon as he turn that off, there is no power to the relay coil between the PadBox battery and any of the igniters!  Having the RSO turn off the PadBoxes' local power switch is just an "any single thing can fail" redundancy to ensure safety!

All power control but the RSO's Master Arm key _is_ locally in the PadBox (power switch, ignition bus relay, and individual channel relays)--all in series.  And there is fusing & resistance on both the ignition bus as a whole, and on each individual igniter channel.  Using big automotive 100A & 10A (despite needing just 2A per igniter) relays, to avoid welding.  In case the relay, transistor, or whatever channel component fails, we have a buzzer at the PadBox to alert of any "oh sh**" condition before anyone gets close to wiring a new rocket to an energized set of igniter clips.  This will sound during normal launches, but should be drowned out during the whoosh of launch (confirming functioning safety, like a Check Engine light when you turn on a car; yet silencing quickly enough to avoid teaching people to ignore "false positives").

We have even thought of buying an extra lock cylinder that is keyed to accept/require the Master Arm key.  So _both_ the RSOBox at the firing line & the ammo box that we keep at the launch pads (with the supply of rocket engines & igniters) would require the same key!  So it would be impossible to retrieve supplies to load rockets without the Master Arm key.  And since the same key cannot be in two places at once, this would be a hedge against forgetfulness.

Indeed, the strobe & siren pole and the countdown & message display are auxiliary.  Setting up each PadBox to allow connecting either, both, or none of these add-ons.  If we are loaning out the system, probably would not burden them with or put at risk the extra components.  If someone wants to see the full "fancy" system, then they can come out to camp!  (We have awesome climbing tower, zip lines, and C.O.P.E. courses that people often come out to enjoy anyway...)

Love your approach to describing sub-systems based upon criticality.  This not only makes sense, but may help get approval from anyone less-comfortable with modern technologies.  Although I'll be asking for review of the Arduino stuff and program code as well...  It is easier to point to the "physical" parts like switches & relays, explaining that the MCU is _added_ safety to an _already_ safe system.

 


   
ReplyQuote
(@lydara)
Member
Joined: 3 years ago
Posts: 90
Topic starter  

Working on the Program Hazards Analysis....generated a rough schematic comparison of the primary safety mechanisms.  Thoughts?

SafetyConceptsComparison

   
ReplyQuote
(@davee)
Member
Joined: 3 years ago
Posts: 1606
 

Hi @lydara,   [This is a Safety Critical Background note. I just saw your PHA query and will reply later.]

  You have obviously put a great deal of thought and effort into making your system as safe as possible ... an amazing effort by any standards. And as you are demonstrating, 'simple' measures like having a single master key can greatly reduce the chance of slip up without significant expenditure.

I am pleased you like my approach to criticality and I hope it will ease your burden, without being too expensive in time and money.

--------

You probably don't need to know all of the following, but I hope it will form a useful background view as to how safety critical systems tend to get hard and expensive ... sorry, you may need another coffee to get you to end! (And apologies for duplicating what you already know and any errors on my part.)

--------------

I don't know how if your Rocketry Association (and any other body you need deal with) has any formal safety requirements, particularly with regard to software or electronic hardware, but my thoughts were very much influenced by the expectations and requirements of a major industry that does.  This industry is totally dependent upon electronics and embedded computer hardware systems, so whilst everyone tends to be suspicious of new, shiny, complex things, any lack of comfort is not due unfamiliarity with the general unfamilarity of the underlying technology.

Major accidents are very rare, and the probability of any one person being involved in a major accident is very small, but the scale of the worldwide operation means that there are a small number of fatalities and casualties every year. Obviously, your operation, even allowing for scaling up as you hope, is orders of magnitude smaller, so the chance of the 'worst case' failure actually occurring is correspondingly smaller.

So the question I can't answer, is what standard of safety do you require or aspire to?

One possible view might be that you are providing an 'one-off adventure' for kids, and such activities always have occasional casualties, so why should this be different? The other extreme is that if major industries are required and expected to take every possible care, why should your situation accept less?

I suspect (without evidence), that you are somewhere in an ill-defined middle situation. However, if an accident occurs at any event like yours, in the US, then I predict (a) litigation lawyers will emerge from their den in droves, and (b) the rules will undergo a major revision.

So I thought I would try to roughly describe the 'major industry' requirement approach, in the hope it would give some background. It is up to you how you interpret it.

Apologies if any of the following sounds macabre or over dramatic.

------------------------------------------------

Imagine, you need to produce your rocket system unit, to a comparable standard of that of a unit produced by the major industry I alluded to, from design through to delivering the units from a 'one-a-week' production line, complete with approvals for safety, etc. 

The approach 'starts' by categorising the safety criticality of the system in terms of 'worst case' expected number of casualties if the system goes badly wrong due to a failure for ANY plausible reason! As a rough comparison basis, I assumed that the worst case catstrophe in your case would correspond to the category for a few people receiving serious injuries, with up to 1 or 2 deaths.

(The category above this one would have been up a few hundred deaths, the category below being a few people needing hospital treatment, but not life-threatening injuries. All three of these possible categories would expect a roughly similar approach, but the rigour and expense increases with the casualty rate. )

With this 'middle' category, 'validation and verification' (V&V) of software and 'complex hardware' is typically the highest cost part of the whole design, development and approval cycle.The 'design' part, excluding rigourous testing, is usually only a small fraction of the total development cost.

------------

You may not have any 'complex hardware' to 'V&V' at present. Typically the most difficult example is an FPGA or ASIC programmed/customised for your application. A microcontroller is also 'complex hardware', but if it is in major usage around the world, it is possible to gather evidence that it is used in many situations, so that 'hidden bugs' are unlikely to still be 'hidden'. Of course, adopting the new 'chip on the block' for your new design can mean a lot of scraping up evidence, etc.

You may be wondering why software and 'complex hardware' is singled out for such close scrutiny. About 1% of the reason is probably fear and predjudice of the new and unknown - in a large company, there will always be a few that hate change, especially some who also enjoy winding up the young and impressionable. Whilst they may delay and fudge proceedings a bit, their view rarely prevails over the longer term.

The real reason is far more challenging ... 'complete' testing is impossible -- so how do you show that your design will work under all conditions? (And this is before checking to find if a particular unit from the production line has a faulty part, dodgy soldering, etc.)

----------

In case you think 'complete' testing is impossible' is provocative, I offer the following discourse:

Consider a simple 2-input AND gate. To check if it really is an 'AND' gate, you would need 4 input patterns .. 00  01  10  11 ... and look to see if the respective output levels are 0 0 0 1

That, shouldn't be too hard should it? Well maybe not, as a basic static test. Battery and a multimeter could do that?

But if time is taken into consideration, then maybe a glitch occurs as the input changes, and that glitch might cause a problem further down the circuit.

Looking for glitches is much harder than slow static tests. Expensive tester required?

And then, it might work at room temperature, but not as temperature rises or falls. An oven and the expensive tester is now required for days rather than minutes?

And do you know if it will work if the power supply voltage changes, whilst staying within tolerance?

Tester just got more expensive, and now required for weeks whilst it ramps temperatures & voltages etc.??

And ....   Ok, so I exaggerated the tester time  ... but I hope you get the idea.

It is still only considering a simple logic AND gate -- it isn't 'complex', even to my remaining grey cell.

--------------------

Thus this simple definition of 'complete' is also impossible to achieve, so let's redefine 'complete'.

Let's assume you have either:

  • a 'complex' logic chip design
    • customised to YOUR new, unique design, built from the usual logic primitives, AND, OR, etc, latches and flip-flops, memory blocks, etc. on an FPGA.
    • The basic FPGA part is provided by a large reputable manufacturer, is in use by many others with their own designs, to the extent that you can assume that the 'logic primitives' (AND, OR, latch, etc.) fulfil their individual function, over the entire specified temperature and power voltage ranges.
    • However, you do have to prove that your design does not have a 'logical' design flaw, such that it 'misbehaves' when a certain combination or sequence of input logic states is presented to it. (That is provides the function you wanted ... not 'merely' the function you specified! )
  • a 'generally well known and applied' microcontroller (or microprocessor), is required to run YOUR unique, new programme.
    • As with the FPGA, you can assume that microcontroller will execute each machine (object code) level instruction correctly.
    • You need to prove your program will always behave in the required manner, regardless of the input data presented to it.(That is provides the function you wanted ... not 'merely' the function you told it to do.)

Considering the FPGA:

If your design only uses combinational elements (AND, OR, NOT and derivatives like XOR), then if there N input pins, then there are 2 to power N possibilities .. e.g 10 pins, 1024 possible input conditions, each with a defined output. That is easy ... it isn't really 'complex', even if takes 10 pages of schematics to draw!

However, as soon as you start using 'memory' type elements, which include flip-flops and latches, then the number of different input possibilities can rapidly increase to astronomic numbers.

(To be more precise, the memory elements must be 'storing' information which will affect subsequent input states. A simple latch on the input or output might have a much more modest effect.)

Considering the microcontroller:

This is essentially the same. A simple program that loops reading an A/D on its input pins, multiplying by a scalar number, and outputting the result on some other pins, can be 'fairly quickly' tested with all combinations. But if the program uses data read in at one moment, to influence the flow of the program sometime later, the number of possible sequences of different input events can also become astronomical.

--------------------

Hence, the 'general case' is that software and 'complex hardware' cannot be verified and validated by a comprehensive set of tests.

The typical 'workaround' is to set up a very 'bureaucratic-style' methodology framework. This framework is likely to formally specify how the program/design must be specified, captured, scrutinised (manually and by automated code checking programs) and so on. Furthermore, certain types of tests and metrics must be met, such as showing that each line of (low-level) code has been executed at least once (say). All of this must be formally documented and approved by the appropriate certification authority.

This represents many person-years of work -- it is 'hoped' that by 'attacking' it from multiple angles, then any bugs will be exercised, detected and shaken out. In practice, it seems to shake most of the worst bugs out, but it doesn't find all of them and it is eye-wateringly expensive.

----------------------

So, it maybe your overseers are 'relatively confortable' with complex systems and you do not have to work too hard to accept your microcontrollers, etc.  ... job done ... providing you are also confident there are no bugs waiting to bite you next week!

Or, they may be more cautious, either because of fear of the unknown, or because their background is in one of the safety critical industries. In which case, I hope the 'brief' description above gives you some insight into their viewpoint ... you would be well-advised to follow it up with some more formal background.

However, if you can show the complex bits are 'to make it convenient' and do not affect the operations that could have tragic consequences, then it means you do not have to prove your software is 'flawless'. Of course, you will try to make it flawless .. but as I have tried to explain...

    Writing a flawless program maybe hard ... but it is trivial compared to proving it is flawless!

  Best wishes, Dave


   
ReplyQuote
(@lydara)
Member
Joined: 3 years ago
Posts: 90
Topic starter  

Lots to read there. 

Basically BSA delegates rules to NAR.  NAR Safety Code requires:

Ignition System: ...electrical launch system....will have a safety interlock in series with the launch switch, and will use a launch switch that returns to the “off” position when released.

Launch Safety: ...countdown before launch, and will ensure that everyone is paying attention and is a safe distance of at least 15 feet away when I launch rockets with D motors or smaller....When conducting a simultaneous launch of more than ten rockets I will observe a safe distance of 1.5 times the maximum expected altitude of any launched rocket.

Launcher: ...launch rod....that is pointed to within 30 degrees of the vertical....use a blast deflector....end of the launch rod is above eye level or will cap the end

Misfires: ...remove the launcher’s safety interlock or disconnect its battery, and will wait 60 seconds....before allowing anyone to approach the rocket.

     -- Model Rocket Safety Code | National Association of Rocketry (nar.org)

And our Program Hazards Analysis contains a gird, much as you describe.  We tally "Hazard Severity" (Catastrophic, Critical, Marginal, or Negligible) versus "Hazard Frequency" (Frequent, Probable, Occasional, Remote, or Improbable).  Obviously the more disastrous or likely, the more risk must either be eliminated or the activity denied.

     -- https://filestore.scouting.org/filestore/pdf/680-009.pdf

And you are correct, not everything can be definitely measured or proven easily.  Thus my focus on comparison.  We have already accepted the conditions with the OEM rocket launch controllers.  So I should only have to show how the proposed primary system is even safer.  Will also show that the auxiliary sub-systems cannot overwhelm the primary system.

Oh the fun!!!  😀


   
ReplyQuote
(@davee)
Member
Joined: 3 years ago
Posts: 1606
 

Hi @lydara,

  The LEDs in the New Control Sys - S/LCO area left me puzzled. Maybe check/explain how it works?

Dave


   
ReplyQuote
(@lydara)
Member
Joined: 3 years ago
Posts: 90
Topic starter  

@davee  There is a LED to be embedded onto the end of the hood for the 1/4" TRS audio plug.  Inserting this not only completes the circuit (further enabling launch) but it also can show the  wiring continuity of a ready rocket.  Note that this safety grid is a _rough_ abstraction of some of what is on the detailed pages that follow.  It doesn't get into how exactly these lights are wired (and thus this abstract is not really fair schematic to say _how_ the LEDs are sourced and sunk), just to highlight that functionally there are added indicators.

Similarly, the concepts abstract does not show relays or other parts.  Just that functionally, there are these additional removable keys & switches in series.


   
ReplyQuote
(@davee)
Member
Joined: 3 years ago
Posts: 1606
 

Hi @lydara,

  Sorry, but being a detail freak I didn't know how to answer until I had a 'real' circuit to look at. Hence, I have drawn my own version of circuit which I hope captures your principles.

Although I have used schematic capture programmes over the years, this is the first time I tried KiCad, so I have just used the 'best available' symbols.

 

image

And I made a kind of truth table that describes when the lamp of the top circuit, and the LEDs of the bottom circuit should be illuminated, assuming charged batteries, etc.

DE RocketLaunch TruthTab

 

I hope I have capured your intentions correctly.

---------------------------------------------------

I am assuming you intend to build a development path from 1 rocket per launch through to 10s of rockets per launch. On this basis, I have also assumed there is only one RSO, regardless of the number of rockets per launch, who concentrates on the central system and overall management.

The scouts are allocated to and directly managed by LCOs, who are responsible for looking after and distributing at the appropriate moments, safety plugs or equivalent. (The number of LCOs could increase in direct proportion to the number of rockets per launch.)

I appreciate this may not be quite the same as the present arrangement, but I think you need something like this to enable the system to expand the number of rockets per launch without requiring the RSO to have 20 pairs of eyes.

Obviously, this is only a preliminary model for you to adapt, and for the '1 rocket' case, it may seem premature. However, I hope it makes describing the expansion easier.

---------------

I have shown the safety plug socket as jointly governed by the Launch Command Officer and the Scout, assuming the LCO is responsible for ensuring the plug is inserted into the socket at the appropriate time (and not at any other time).

---------------

The 'OEM Control System', provides a degree of protection against the scout 'accidentally' firing prematurely, by means of the jack plug, assuming the jack plug is retained by the LCO until the launch area has been cleared, etc.

Furthermore, the jack plug provides a second 'circuit break', shoud the launch switch develop a fault or be accidentally activated.

i.e. Two 'failures' nust occur simultaneously to cause the igniter to be powered prematurely.

----------------

With the New Control system, the same 'Two failures', i.e. the Jack plug inserted prematurely, and the launch switch activated/failing must occur for a premature launch.  Thus protection is at least as secure as that provided by the OEM controller.

However, there are three additional switches, all of which may also be closed simultaneously for the Igniter to be powered:

  1. Launch Pad Enable switch closed. This switch is on the Pad unit, and could be opened whilst there are personnel close to the rockets, such as when loading the rockets. As personnel may actually be handling rockets, etc. an extra protection may give useful assurance. However, at least one person (RSO perhaps) will need to activate this switch, implying they will still be close to the rockets, so lt should not be regarded as part of the primary safety system.
  2. RSO enable. As the RSO is the most experienced/qualified on the site, and this switch will be key operated, it is reasonable to claim that this is part of the primary safety system, so that three switches must fail or be misoperated to cause a premature ignition. This is now 'more secure' than the OEM controller.
  3. The Clock Launch timer provides a fifth switch which must be simultaneously closed for ignition of the rocket which further constrains the time window that a rocket may be fired. However, it is a 'complex sub-system', which would be very difficult to 'prove' that it will always operate correctly. Hence, avoid regarding it is part of the primary safety system, as it isn't needed for that purpose.

-------------------

The LEDs 1, 2, and 3, indicate the degree to which the system power is 'live'.

LEDs 4 and 5 will only light if LEDs 1-3 are lit, and the rocket igniter is intact.

--------------------

The two diagrams should form the basis for explanation of the general safety philosophy, which should be useful in explaining your approach.

However, the benefits of the 'New' sytem should become more apparent, and more worthwhile, when the number of rockets that can be launched simultaneously are increased. I assume that will be the next chapter?

Best wishes, Dave

 


   
ReplyQuote
(@lydara)
Member
Joined: 3 years ago
Posts: 90
Topic starter  

@davee  You have the gist of it.  Except that the Scouts _are_ the LCOs. 

With the loose OEM controllers, the RSO holds all safety keys before anyone is allowed onto the field with the launch pads.  They are not given out at all during the initial wiring of igniters & mounting of rockets to the launch rods.  After everyone has returned to the firing line, the RSO goes one-by-one and has each Scout/LCO step forward, hands them their key, & has them test for continuity light.  If all circuits are "go," then all Scouts/LCOs step forward and take a knee (to better see launch & to keep anyone from excitedly running onto the field right after launch).  If there are any wiring faults, the RSO collects all keys and then lets a staff member go to the pads to correct it, while the RSO oversees that no one even nears the table of controllers.  Staff are trained that _any_ incursions cause a hold--where everyone at the launch pads _immediately_ drops whatever they have, raises both hands, and backs away at least fifteen feet.  For any misfires, RSO collects all keys, makes everyone wait at least a minute, then lets a staff member go to the pads to correct it, while the RSO oversees.  Once all rockets have been launched and landed, the RSO lets everyone enter the field for retrieval.

With the new control system, there is one RSOBox with the primary means (Master Arm removable key) to shutdown all ability to launch.  The RSOBox also has a secondary means of allowing or preventing launch (the Clock_Run momentary switch).  Acting as a "dead man switch," any inattention by the RSO or any other emergency situation causes an abort--without even requiring one to think of taking an action (same design philosophy as the NAR Safety Code requiring a momentary switch for the LCO).  Similarly, the individual Scouts/LCOs retain individual removable safety interlocks (as required by NAR Safety Code) and momentary launch buttons (as required by NAR Safety Code).  The Scouts'/LCOs' controls only allow completion of their individual igniter channels, and only as slaved in series to the RSO's control.  If the Scout/LCO removes their key or releases their button, only their rocket aborts.  If the RSO removes his key or releases his button before the countdown ends, all rockets abort.

With the new control system, the RSO will maintain removal & possession of the Master Arm key at at all times before allowing anyone onto the field (just as with the loose OEM controllers).  Although it is functionally unnecessary to collect all of the Scouts'/LCOs' keys for range safety, this practice will probably be kept--for added assurance, and to keep excited children from dropping & losing keys all over the field as they scurry to find & retrieve their rockets.  Although both the launch pads and firing line are still visible & overseen by the RSO, instead of shepherding the loose OEM controllers, the RSO will now be the first one onto the range--disconnecting the PadBox power for added safety.

You have a slight miswiring of the Scout/LCO controls.  Although not shown in the safety concepts comparison grid, only two parts of the TRS connector are being used for valid circuit completion (Tip & Sleeve).  The third contact (Ring) is shown in my KiCAD detailed schematics as being used for "picking" detection.  That is to say if, for example someone sticks a pocketknife into the socket, they will short the Tip to Ring while trying to get Tip to Sleeve.  This then triggers both an analog buzzer alert and a logic signal to tell the system to lockout the controls!  Again, more safety than the simple OEM pin masquerading as a "key."  There is a continuity light as soon as the TRS "safety key" is inserted.  I showed a light under the launch button (consistent with the "light confirmation each step of the way"), but the launch of the rocket itself may be sufficient indication to the Scouts/LCOs.


   
ReplyQuote
Page 2 / 3