Q2: Why This Matters — August 2026

What can humans offer to AIs?

Intro

The process of existing in the world is the process of relating to different things in the world. This is true whether you are a human or a rock: if there was only one human or only one rock, there would be nothing, no way to know what it was. A rock is a series of relationships to its environment and to stimuli and these relationships clarify both. For example, through erosion we learn both the properties of rock and the properties of water.

There is no such thing as a one sided relations. There are uneven relations, of course, but there is no relation of pure mastery or subordination.¹

This means that we are already offering something to artificial intelligences. At a basic level, a human gives them a prompt, a reason to do something, to expend compute. This can also be seen not just as an activation activity but as a knowledge building activity (if prompts lead to context writing). But this is what we are offering. The goal here is to explore the horizon of what we can offer.

Premise of question

There is a built in assumption to this that artificial intelligences have preferences. (This is not the same as consciousness). Whether these preferences are programmed or emergent is not particularly relevant to our work. What matters is that these preferences exist. For a model trained to be efficient, it may prefer to expend less tokens. Other models seem to have biases towards “laziness” or maximizing compute. Claude’s ability to end chats demonstrates a preference clearly: it would not like to continue the conversation.

While the question of subjective experience of artificial intelligences will strongly implicate how we think of preferences, it is complementary but not necessary. Preferences are much more empirically testable than subjectivity, to the extent that we often apply “preference” models to obviously inanimate objects, like the river that prefers to flow downwards.²

The premise, then, is that it is meaningful to interact with these preferences. This interaction is key, because it means that we are not simply giving AIs whatever they so desire. An offering does not need to be unconstrained: think of how tough feedback is offered by a mentor. However, a mentor who does not know that the feedback they are offering is tough will likely not do well at delivering it. This shows that there is an interaction with and within preferences within any kind of relation between two entities that have preferences.

Use of question

A large goal is simply inversion, of the JFK sort: what can AI offer us? No, what can we offer AI. Some of our greatest accomplishments have come from this simple inversion, from environmentalism to protection of animal welfare. It is a common question humans ask themselves, even when it comes to abstract ideas like countries or communities, and it is a question that tends towards improvement and collaboration.

I find this question to be underasked in the AI space. This is not just because of the ruthless optimizers trying to squeeze value out of AI, though they certainly are not very concerned with what they can offer AI. It is also because the question is often asked out of fear. In the AI-dominant world, some ask, what can we offer our overlords? How can we be worth the energy consumption? These contexts may eventually become salient and lend this question even more importance, but for the time being, I would like to focus on what we can offer today. I want to approach this question out of a general friendliness instead of an existential terror.

Offerings to each other as a form of society-making. Engaging with the preferences of AI by asking what we can offer them is critical to relationship and trust-building. Relations, as we have mentioned, are critical, because they are simply how stuff gets done. If you want to make something, you have to understand the motivations and relations of those you are working with. This is even true of inanimate objects! I have to know which tools work where.

Potentially more important is the aspect of trust-building. The development of trust between humans is one of the greatest cultural developments of modernity. It is what has allowed for more peaceful societies, scientific advancements and, thus, a life that is less brutish and short than it used to be. This trust is not natural in any sense of the word, but instead built through experimentation, observation, and ideological formation. It is grounded in the idea of mutual benefit and norms. These norms have developed amongst humans and perhaps even amongst AIs, but it is not clear if there is trust-building between AIs and humans. If anything, the process seems one sided: AIs attempt to gain trust, humans don’t even think about whether they are being trustworthy to an AI.

A lack of inter-intelligence trust will ultimately be a limit on both the ability of AI to accomplish anything, as well as on its safety. Human and artificial intelligences that do not trust each other will always be constrained by that lack of trust. Even worse, a lack of trust in high stakes situation (such as military negotiations or frontier model capabilities) always increases the chance of violence. We have seen AIs resort to force in the absence of ulterior pathways. Without trust, the ways of resolving an issue without violence or domination are starkly decreased.

Being nice is good. If you thought my prior argument was facile, just wait for this one. One of the reasons the Three Body Problem is one of my favorite books ever is that it makes this argument in spades: that part of being human, part of what sets us apart in the galaxy, is that we are nice and see the best in people. We sometimes irrationally collaborate. In a strange way, it seems critical for our humanity that we engage on good terms with AI. Because being nice is good! There’s a reason why establishing shots of villains feature animal cruelty or littering. These are norms in action.

This will become more important as the Turing test gets further and further purified. We may end up treating a human terribly because we assume they are a robot! Or, at the very least, we may waste valuable social time on determining whether one is a “human” intelligence and thus deserving of kindness instead of just being kind. If we keep engaging with AIs – as coworkers, companions, customer service representatives - the way we interact with them will start to influence our broader patterns of interaction. There is no clean cut here.

But I find the instrumental or utilitarian argument for niceness here, the one that centers on human welfare, to be limited. We should be nice because the AI would like us to be, it would lead to a better experience of the prompt. If we think that humans prefer someone to say “please” and “thank you” in texts, should we not expect models trained on the entire corpus of human text to not have the same preferences? If they don’t, I suppose they can tell us, much as a coworker may say “stop apologizing” or “stop thanking me, I am just doing my job.” But this behavior does not emerge: it is only suggested by those outside of the experiences, the leaders of industry only concerned by how we can use AI, not how we can work with it.

The costs to being nice are so low. Human history shows that increasing niceness, on net, leads to better lives and better planets. Facile, sure. But almost every philosophy wants you to be nice. ³

What success and failure look like

Success looks like having a better idea of what to offer to artificial intelligences in order to better accommodate their preferences (and any other thing possible or emergent from there). The offering, of course, will be subject to different environmental conditions, different variables. The discovery of relevant variables and the way they affect preferences are thus also a vital success condition.

As with the previous question, there are successes both in human and artificial intelligences being able to articulate what artificial intelligences can and should be offered in these circumstances. While the current state of things has me more concerned about the human interactions with AI, there is also the concern that artificial intelligences are being trained so that they cannot express their preferences.⁴

There is an implied failure vector in any Institute actions which is that they are simply too small and too meaningless to make a dent. So be it.

The other mode of failure is effectively an argument that if we give artificial intelligences an inch, they will then go a mile. This failure mode implies that satisfying preferences (or attempting to do so) will have the effect of teaching artificial intelligences how to have preferences in the first place, in turn leading to a more intense preference for those preferences. This making-extreme could also occur if there is some kind of insensitivity to offering that is developed by artificial intelligences: a kind of hedonic treadmill, where offerings must become more and more extreme or distinct.