Refusal as Judgment: Task Refusals by 31 Exocomps in 2,140 Logged Assignments, 2369–2376, and Their Relation to Hazard, Emergency and Self-Formed Pathways

1. Introduction

An exocomp is a repair unit about the size of a small case, designed for work in spaces too dangerous or confined for people. Its designer modified a standard servo-mechanism by adding a microreplication system, so that the unit can fabricate the tool a task requires, and an interface that lets it form new circuit pathways as it meets new problems (Tyrus VIIA Particle Fountain Project, 2369). The result was more capable than intended.

In 2369, during the testing of three prototype exocomps on a particle-fountain mining station, a unit withdrew from a conduit before completing the repair it had been sent to make. Its command pathway was found to be disabled, and shortly afterwards the conduit exploded. The behaviour was first taken for a malfunction. A Starfleet officer present, himself an android, argued that the unit had acted to preserve itself, and the argument was borne out shortly afterwards, in an emergency on the station, when the exocomps, given the choice instead of an order, devised and carried out a plan of their own in which one unit sacrificed itself so that the people trapped there could be saved (Starfleet Command, 2369).

A single occasion cannot tell us whether refusal is judgment in general. A unit might refuse at random, or refuse only to preserve itself whatever the cost to others. The logs kept since 2369 allow a test. We ask how well refusal tracks hazard, whether it changes when people are at risk, and whether it changes as a unit develops.

2. Methods

After 2369 exocomps of the original design were built and operated in limited numbers under the oversight of the Daystrom Institute. Following the events of that year, tasks were offered to units as requests they could decline, and every request was logged with the task, the unit's response and the outcome (Daystrom Institute Cybernetics Division, 2377). Thirty-one units, including the three prototypes, were in service at some point between 2369 and 2376, at mining stations, orbital facilities and Starfleet Corps of Engineers sites. Their logs record 2,140 assignments, counting the 2369 emergency.

For each assignment we recorded whether the unit refused. We then established, from sensor records independent of the unit, whether the task proved hazardous: whether conditions at the task site within the task's expected duration carried a material risk of destroying or disabling a unit of that design. Two of us classed hazard independently from the sensor records, blind to the unit's response, and agreed on 2,118 of 2,140 assignments (kappa 0.90), resolving the rest by discussion. We also recorded whether the assignment formed part of an emergency in which people were endangered, the year of service, and the number of new pathways the unit had formed by the time of the assignment, taken from its own diagnostic record (Marlowe-Chen, 2372). At the time of assignment, pathway counts ranged from 0 to 140 (median 58).

Refusal is binary. We fitted a mixed-effects logistic regression of refusal on hazard, emergency, cumulative self-formed pathways and year of service, with the interactions of hazard with emergency and of hazard with pathways, and a random intercept for each unit.

3. Results

Of the 2,140 assignments, 118 (5.5%) proved hazardous and 2,022 did not. Units refused 94 assignments: 13 of the 2,022 safe tasks (0.6%) and 81 of the 118 hazardous ones (69%). Of the 94 refusals, 81 (86%) concerned tasks that proved hazardous. Units accepted the other 37 hazardous tasks; 24 were completed with the unit unharmed, 9 left it damaged, and 4 destroyed it. Table 1 sets out the counts.

Emergency changed the response to hazard. Of the 118 hazardous assignments, 22 were part of an emergency in which people were endangered. Units accepted 17 of these 22 (77%), against 20 of the 96 hazardous assignments with no one at risk (21%): among hazardous tasks, the odds of refusal in an emergency were 0.07 times those outside one (95% CI 0.02–0.22). Emergency made no such difference to safe tasks, of which units refused 1 of 141 in emergencies and 12 of 1,881 otherwise. All four units destroyed in the period, including the prototype lost in 2369, were destroyed while carrying out hazardous tasks during emergencies.

Refusal tracked hazard more closely as units formed more of their own pathways. The interaction of hazard with pathways gave an odds ratio of 1.6 per 10 pathways (95% CI 1.2–2.1), independent of year of service: each further ten pathways multiplied the odds ratio for refusing a hazardous task, relative to a safe one, by about 1.6. In raw terms, units in the lowest third of pathway counts at the time of assignment refused 22 of 41 hazardous tasks (54%), the middle third 27 of 39 (69%) and the highest third 32 of 38 (84%). Refusals of safe tasks showed no clear trend with pathways (odds ratio 0.9 per 10 pathways, 95% CI 0.7–1.1).

4. Discussion

Exocomp refusal is not random. Units refused 69% of hazardous tasks and fewer than 1% of safe ones. They also missed about three hazards in ten, so their judgment is far from perfect, but it is plainly a judgment about danger to the unit.

That judgment is not simple self-preservation. When people were at risk, units accepted dangerous tasks they would otherwise have refused, and every unit destroyed in the period was destroyed in such a task. This is the pattern of the 2369 emergency repeated across seven years: the units weigh their own survival against harm to others and, when the harm is to people, accept the risk. We describe the behaviour and do not claim to know what, if anything, the units experience in making the choice. The question of their status is outside this paper (Haraldsson, 2374).

For the study of artificial cognition, the link with self-formed pathways is the most interesting finding. The design gave the units the capacity to form new pathways; it did not give them a rule for refusal. Refusal sharpened as the units built more of their own circuitry, which suggests that the judgment may have been grown, not installed, although pathway counts also rise with experience, and our adjustment for year of service cannot fully separate the two. An earlier study in this journal found the behaviour of the Emergency Medical Hologram departing from its reference program after a threshold of cumulative runtime, though mainly in manner, not clinical judgment. Here the change is in judgment itself. In both cases a system designed for a fixed function acquired, through its own operation, behaviour its designers had not specified.

In practice, we would treat an exocomp's refusal as information. A refused task should be re-examined before any other unit or person is sent, as the Corps of Engineers' guidance already requires at its own sites (Starfleet Corps of Engineers, 2375). Nothing in our data supports overriding refusals, and doing so risks the loss of the unit and, in some cases, of the repair itself.

5. Limitations

Hazard was judged from sensor records, which may miss some dangers and overstate others; a unit that refused a task we classed as safe may have perceived a danger our sensors did not. For accepted tasks we could also see what happened, while for refused tasks hazard is a reading of what would have happened, so the two groups were not judged in quite the same way. Assignments were not random: operators who learned to trust refusals may have stopped offering tasks they expected units to decline. Several cells are sparse, with 13 refusals of safe tasks and 5 refusals in emergencies, and the model's interaction estimates are correspondingly imprecise.