Resilient post-step hardware cleanup on self-hosted runners when Modbus socket drops during edge I/O tests

[Tekhabeeb]

🏷️ Discussion Type

Question

💬 Feature/Topic Area

API

Body

Hi all,

We are setting up an automated hardware-in-the-loop (HIL) testing pipeline using GitHub Actions to validate firmware updates and access microservices on our physical perimeter infrastructure.uld be very helpful.

Our edge test rig consists of an industrial Raspberry Pi CM4 gateway (running Ubuntu Server 24.04 LTS, kernel 6.8.0-1015-raspi) configured as a self-hosted runner. As part of executing integration tests against our on-prem gate barrier system controller interface, the workflow sends automated pulse and hold commands via Modbus TCP (port 502) to verify loop detector status and relay trip intervals.

Workflow Configuration (.github/workflows/edge-hardware-ci.yml):

name: Edge Barrier Actuation Test
on:
  workflow_dispatch:
    inputs:
      actuation_hold_ms:
        description: 'Relay dwell duration in ms'
        required: true
        default: '2500'
jobs:
  validate-barrier-io:
    runs-on: [self-hosted, edge-gateway-linux]
    timeout-minutes: 5
    steps:
      - name: Checkout Suite
        uses: actions/checkout@v4
      - name: Execute Relay Trip Cycle
        id: trip_test
        run: |
          python3 scripts/hw_trip_harness.py \
            --host 192.168.20.110 \
            --dwell ${{ github.event.inputs.actuation_hold_ms }} \
            --enforce-failsafe
      - name: Hard Hardware Failsafe Reset
        if: always()
        run: |
          echo "Executing hardware failsafe reset coil pulse..."
          python3 scripts/emergency_release.py --force-all-coils-low

Questions for the Community:

1. Is relying on ACTIONS_RUNNER_HOOK_JOB_COMPLETED environment hooks on the self-hosted runner considered the standard pa
ttern for physical I/O reset scripts, or can that hook also be skipped during abrupt runner disconnects?

2. How do teams doing embedded/hardware CI with self-hosted runners handle hardware watchdogs to ensure physical relays revert to a safe de-energized state if the runner process dies mid-step?

[HIRATRA]

I would treat GitHub Actions cleanup as a best-effort recovery layer, not as the primary safety mechanism for physical I/O.

For your first question, ACTIONS_RUNNER_HOOK_JOB_COMPLETED runs at the end of a job, after its workflow steps. It is useful as a runner-level cleanup fallback, but it cannot guarantee execution if the runner host loses power, the runner process is killed, or the machine becomes unable to execute the hook. Likewise, if: always() only controls step scheduling within a running workflow; it cannot recover from every host-level failure. GitHub documents the hook behavior here: https://github.com/proxy/docs.github.com/en/actions/how-tos/manage-runners/self-hosted-runners/run-scripts

For a HIL setup like yours, I'd use several independent layers:

1. Hardware/controller watchdog: The controller should automatically return outputs to a defined safe state if it stops receiving a heartbeat or valid commands within a bounded timeout. This must operate independently of the GitHub runner.
2. Bounded actuation: Make relay activation time-limited at the controller or hardware layer, rather than relying on Python to send the eventual OFF command. A process crash should not leave a coil energized indefinitely.
3. Layered cleanup: Keep the if: always() step for normal failures, and consider the runner completion hook as an additional fallback. Neither should be the safety guarantee.
4. Recovery verification: After a test, read back the relevant output state and fail the job if the expected safe state cannot be confirmed. If communication is lost, report the state as unknown rather than assuming the reset succeeded.
5. Fault-injection tests: Test runner termination, network loss, controller disconnection, and power interruption in a controlled environment to verify that the hardware reaches its safe state without GitHub Actions.

One important nuance: “all coils low” is only a safe state if the equipment's control and safety design explicitly defines it as such. For an actual gate or barrier, the safe response depends on the mechanism and hazard analysis; software-level Modbus commands should not replace appropriately designed safety interlocks.

The key architectural question is: if the Raspberry Pi loses power immediately after sending the relay ON command, what independent mechanism guarantees that the output returns to a safe state? If the answer depends on another GitHub Actions step, the system still has a single point of failure.