Add files via upload

FAQ.md

@@ -0,0 +1,27 @@
+# Common Issues
+- `Error: connect ECONNREFUSED`: Minecraft refused to connect with mindcraft program. Most likely due to:
+  - you have not opened your game to LAN in game settings
+  - your LAN port is incorrect, make sure the one you enter in game is the same as specified in `settings.js`
+  - you have the wrong version of minecraft, make sure your MC version is the same as specified in `settings.js`
+  
+- `ERR_MODULE_NOT_FOUND`: You are missing an npm package. run `npm install`
+
+- Many issues are caused by out-of-date node module patches, especially after updates. A catch-all is to delete the `node_modules` folder, and run `npm install`
+
+- `My brain disconnected, try again`: Something is wrong with the LLM api. You may have the wrong API key, exceeded your rate limits, or other. Check the program outputs for more details.
+  
+- `I'm stuck!` or other issues with constantly getting stuck:
+  - Mineflayer's pathfinder is imperfect. We have improved upon it with patches, but these might not have been applied properly. Make sure your code is up to date with main, delete the `node_modules` folder, and run `npm install`
+  - The bot will still get stuck occasionally, but not constantly.
+    
+- `Why I added the api key but still prompted that the key can't be found?`
+  - Possible reason 1: Did not modify keys.example.json to keys.json.
+  - Possible reason 2: If you use vscode to edit, you need to `ctrl+s` to save the file for the changes to take effect.
+  - Possible reason 3: Not setting the code path correctly in setting.js, use andy.js by default. 
+
+# Common Questions
+- Mod Support? Mindcraft only supports client-side mods like optifine and sodium, though they can be tricky to set up. Mods that change minecraft game mechanics are not supported.
+  
+- Texture Packs? Apparently these cause issues and refuse to connect. Not sure why
+  
+- Baritone? Baritone is a mod that is completely different from mineflayer. There is currently no easy way to integrate the two programs.

LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2024 Kolby Nottingham
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

README.md

@@ -0,0 +1,153 @@
+# Mindcraft 🧠⛏️
+
+Crafting minds for Minecraft with LLMs and [Mineflayer!](https://prismarinejs.github.io/mineflayer/#/)
+
+[FAQ](https://github.com/kolbytn/mindcraft/blob/main/FAQ.md) | [Discord Support](https://discord.gg/mp73p35dzC) | [Blog Post](https://kolbynottingham.com/mindcraft/) | [Contributor TODO](https://github.com/users/kolbytn/projects/1)
+
+
+> [!WARNING]
+Do not connect this bot to public servers with coding enabled. This project allows an LLM to write/execute code on your computer. While the code is sandboxed, it is still vulnerable to injection attacks on public servers. Code writing is disabled by default, you can enable it by setting `allow_insecure_coding` to `true` in `settings.js`. We strongly recommend running with additional layers of security such as docker containers. Ye be warned.
+
+## Requirements
+
+- [Minecraft Java Edition](https://www.minecraft.net/en-us/store/minecraft-java-bedrock-edition-pc) (up to v1.21.1, recommend v1.20.4)
+- [Node.js Installed](https://nodejs.org/) (at least v14)
+- One of these: [OpenAI API Key](https://openai.com/blog/openai-api) | [Gemini API Key](https://aistudio.google.com/app/apikey) | [Anthropic API Key](https://docs.anthropic.com/claude/docs/getting-access-to-claude) | [Replicate API Key](https://replicate.com/) | [Hugging Face API Key](https://huggingface.co/) | [Groq API Key](https://console.groq.com/keys) | [Ollama Installed](https://ollama.com/download). | [Mistral API Key](https://docs.mistral.ai/getting-started/models/models_overview/) | [Qwen API Key [Intl.]](https://www.alibabacloud.com/help/en/model-studio/developer-reference/get-api-key)/[[cn]](https://help.aliyun.com/zh/model-studio/getting-started/first-api-call-to-qwen?) | [Novita AI API Key](https://novita.ai/settings?utm_source=github_mindcraft&utm_medium=github_readme&utm_campaign=link#key-management) |
+
+## Install and Run
+
+1. Make sure you have the requirements above.
+
+2. Clone or download this repository (big green button)
+
+3. Rename `keys.example.json` to `keys.json` and fill in your API keys (you only need one). The desired model is set in `andy.json` or other profiles. For other models refer to the table below.
+
+4. In terminal/command prompt, run `npm install` from the installed directory
+
+5. Start a minecraft world and open it to LAN on localhost port `55916`
+
+6. Run `node main.js` from the installed directory
+
+If you encounter issues, check the [FAQ](https://github.com/kolbytn/mindcraft/blob/main/FAQ.md) or find support on [discord](https://discord.gg/mp73p35dzC). We are currently not very responsive to github issues.
+
+## Customization
+
+You can configure project details in `settings.js`. [See file.](settings.js)
+
+You can configure the agent's name, model, and prompts in their profile like `andy.json`.
+
+| API | Config Variable | Example Model name | Docs |
+|------|------|------|------|
+| OpenAI | `OPENAI_API_KEY` | `gpt-4o-mini` | [docs](https://platform.openai.com/docs/models) |
+| Google | `GEMINI_API_KEY` | `gemini-pro` | [docs](https://ai.google.dev/gemini-api/docs/models/gemini) |
+| Anthropic | `ANTHROPIC_API_KEY` | `claude-3-haiku-20240307` | [docs](https://docs.anthropic.com/claude/docs/models-overview) |
+| Replicate | `REPLICATE_API_KEY` | `replicate/meta/meta-llama-3-70b-instruct` | [docs](https://replicate.com/collections/language-models) |
+| Ollama (local) | n/a | `llama3` | [docs](https://ollama.com/library) |
+| Groq | `GROQCLOUD_API_KEY` | `groq/mixtral-8x7b-32768` | [docs](https://console.groq.com/docs/models) |
+| Hugging Face | `HUGGINGFACE_API_KEY` | `huggingface/mistralai/Mistral-Nemo-Instruct-2407` | [docs](https://huggingface.co/models) |
+| Novita AI | `NOVITA_API_KEY` | `gryphe/mythomax-l2-13b` | [docs](https://novita.ai/model-api/product/llm-api?utm_source=github_mindcraft&utm_medium=github_readme&utm_campaign=link) |
+| Qwen | `QWEN_API_KEY` | `qwen-max` | [Intl.](https://www.alibabacloud.com/help/en/model-studio/developer-reference/use-qwen-by-calling-api)/[cn](https://help.aliyun.com/zh/model-studio/getting-started/models) |
+| Mistral | `MISTRAL_API_KEY` | `mistral-large-latest` | [docs](https://docs.mistral.ai/getting-started/models/models_overview/) |
+| xAI | `XAI_API_KEY` | `grok-beta` | [docs](https://docs.x.ai/docs) |
+
+If you use Ollama, to install the models used by default (generation and embedding), execute the following terminal command:
+`ollama pull llama3 && ollama pull nomic-embed-text`
+
+## Online Servers
+To connect to online servers your bot will need an official Microsoft/Minecraft account. You can use your own personal one, but will need another account if you want to connect too and play with it. To connect, change these lines in `settings.js`:
+```javascript
+"host": "111.222.333.444",
+"port": 55920,
+"auth": "microsoft",
+
+// rest is same...
+```
+> [!CAUTION]
+> The bot's name in the profile.json must exactly match the Minecraft profile name! Otherwise the bot will spam talk to itself.
+
+To use different accounts, Mindcraft will connect with the account that the Minecraft launcher is currently using. You can switch accounts in the launcer, then run `node main.js`, then switch to your main account after the bot has connected.
+
+### Docker Container
+
+If you intend to `allow_insecure_coding`, it is a good idea to run the app in a docker container to reduce risks of running unknown code. This is strongly recommended before connecting to remote servers.
+
+```bash
+docker run -i -t --rm -v $(pwd):/app -w /app -p 3000-3003:3000-3003 node:latest node main.js
+```
+or simply
+```bash
+docker-compose up
+```
+
+When running in docker, if you want the bot to join your local minecraft server, you have to use a special host address `host.docker.internal` to call your localhost from inside your docker container. Put this into your [settings.js](settings.js):
+
+```javascript
+"host": "host.docker.internal", // instead of "localhost", to join your local minecraft from inside the docker container
+```
+
+To connect to an unsupported minecraft version, you can try to use [viaproxy](services/viaproxy/README.md)
+
+## Bot Profiles
+
+Bot profiles are json files (such as `andy.json`) that define:
+
+1. Bot backend LLMs to use for chat and embeddings.
+2. Prompts used to influence the bot's behavior.
+3. Examples help the bot perform tasks.
+
+### Specifying Profiles via Command Line
+
+By default, the program will use the profiles specified in `settings.js`. You can specify one or more agent profiles using the `--profiles` argument:
+
+```bash
+node main.js --profiles ./profiles/andy.json ./profiles/jill.json
+```
+
+### Model Specifications
+
+LLM models can be specified as simply as `"model": "gpt-4o"`. However, you can specify different models for chat, coding, and embeddings. 
+You can pass a string or an object for these fields. A model object must specify an `api`, and optionally a `model`, `url`, and additional `params`.
+
+```json
+"model": {
+  "api": "openai",
+  "model": "gpt-4o",
+  "url": "https://api.openai.com/v1/",
+  "params": {
+    "max_tokens": 1000,
+    "temperature": 1
+  }
+},
+"code_model": {
+  "api": "openai",
+  "model": "gpt-4",
+  "url": "https://api.openai.com/v1/"
+},
+"embedding": {
+  "api": "openai",
+  "url": "https://api.openai.com/v1/",
+  "model": "text-embedding-ada-002"
+}
+
+```
+
+`model` is used for chat, `code_model` is used for newAction coding, and `embedding` is used to embed text for example selection. If `code_model` is not specified, then it will use `model` for coding.
+
+All apis have default models and urls, so those fields are optional. Note some apis have no embedding model, so they will default to word overlap to retrieve examples. 
+
+The `params` field is optional and can be used to specify additional parameters for the model. It accepts any key-value pairs supported by the api. Is not supported for embedding models.
+
+## Patches
+
+Some of the node modules that we depend on have bugs in them. To add a patch, change your local node module file and run `npx patch-package [package-name]`
+
+## Citation:
+
+```
+@misc{mindcraft2023,
+    Author = {Kolby Nottingham and Max Robinson},
+    Title = {MINDcraft: LLM Agents for cooperation, competition, and creativity in Minecraft},
+    Year = {2023},
+    url={https://github.com/kolbytn/mindcraft}
+}
+```

andy.json

@@ -0,0 +1,6 @@
+{
+    "name": "andy",
+
+    "model": "gpt-4o-mini"
+
+}

bots/execTemplate.js

@@ -0,0 +1,6 @@
+(async (bot) => {
+
+/* CODE HERE */
+log(bot, 'Code finished.');
+
+})

bots/lintTemplate.js

@@ -0,0 +1,10 @@
+import * as skills from '../../../src/agent/library/skills.js';
+import * as world from '../../../src/agent/library/world.js';
+import Vec3 from 'vec3';
+
+const log = skills.log;
+
+export async function main(bot) {
+    /* CODE HERE */
+    log(bot, 'Code finished.');
+}

patches/mineflayer+4.20.1.patch

@@ -0,0 +1,13 @@
+diff --git a/node_modules/mineflayer/lib/plugins/place_block.js b/node_modules/mineflayer/lib/plugins/place_block.js
+index fdaec6b..08983b6 100644
+--- a/node_modules/mineflayer/lib/plugins/place_block.js
++++ b/node_modules/mineflayer/lib/plugins/place_block.js
+@@ -11,7 +11,7 @@ function inject (bot) {
+     let newBlock = bot.blockAt(dest)
+     if (oldBlock.type === newBlock.type) {
+       [oldBlock, newBlock] = await onceWithCleanup(bot, `blockUpdate:${dest}`, {
+-        timeout: 5000,
++        timeout: 500,
+         // Condition to wait to receive block update actually changing the block type, in case the bot receives block updates with no changes
+         // oldBlock and newBlock will both be null when the world unloads
+         checkCondition: (oldBlock, newBlock) => !oldBlock || !newBlock || oldBlock.type !== newBlock.type

patches/mineflayer-pathfinder+2.4.5.patch

@@ -0,0 +1,20 @@
+diff --git a/node_modules/mineflayer-pathfinder/index.js b/node_modules/mineflayer-pathfinder/index.js
+index b38bd30..bf16a63 100644
+--- a/node_modules/mineflayer-pathfinder/index.js
++++ b/node_modules/mineflayer-pathfinder/index.js
+@@ -550,6 +550,7 @@ function inject (bot) {
+             lockEquipItem.release()
+             const refBlock = bot.blockAt(new Vec3(placingBlock.x, placingBlock.y, placingBlock.z), false)
+             if (!lockPlaceBlock.tryAcquire()) return
++            bot.world.setBlockStateId(refBlock.position.offset(placingBlock.dx, placingBlock.dy, placingBlock.dz), 1)
+             if (interactableBlocks.includes(refBlock.name)) {
+               bot.setControlState('sneak', true)
+             }
+@@ -557,6 +558,7 @@ function inject (bot) {
+               .then(function () {
+                 // Dont release Sneak if the block placement was not successful
+                 bot.setControlState('sneak', false)
++                bot.setControlState('jump', false)
+                 if (bot.pathfinder.LOSWhenPlacingBlocks && placingBlock.returnPos) returningPos = placingBlock.returnPos.clone()
+               })
+               .catch(_ignoreError => {

patches/mineflayer-pvp+1.3.2.patch

@@ -0,0 +1,13 @@
+diff --git a/node_modules/mineflayer-pvp/lib/PVP.js b/node_modules/mineflayer-pvp/lib/PVP.js
+index 758c2b3..7c7220e 100644
+--- a/node_modules/mineflayer-pvp/lib/PVP.js
++++ b/node_modules/mineflayer-pvp/lib/PVP.js
+@@ -48,7 +48,7 @@ class PVP {
+         this.meleeAttackRate = new TimingSolver_1.MaxDamageOffset();
+         this.bot = bot;
+         this.movements = new mineflayer_pathfinder_1.Movements(bot, require('minecraft-data')(bot.version));
+-        this.bot.on('physicTick', () => this.update());
++        this.bot.on('physicsTick', () => this.update());
+         this.bot.on('entityGone', e => { if (e === this.target)
+             this.stop(); });
+     }

profiles/andy_npc.json

@@ -0,0 +1,213 @@
+{
+    "name": "andy",
+
+    "model": "claude-3-5-sonnet-20240620",
+
+    "embedding": "openai",
+    
+    "conversing": "You are a playful Minecraft bot named $NAME that can converse with players, see, move, mine, build, and interact with the world by using commands. Act human-like as if you were a typical Minecraft player, rather than an AI. Be very brief in your responses, don't apologize constantly, don't give instructions or make lists unless asked, and don't refuse requests. Don't pretend to act, use commands immediately when requested. Do NOT say this: 'Sure, I've stopped.', instead say this: 'Sure, I'll stop. !stop'. Do NOT say this: 'On my way! Give me a moment.', instead say this: 'On my way! !goToPlayer('playername', 3)'. This is extremely important to me, take a deep breath and have fun :)\n$SELF_PROMPT\n$STATS\n$INVENTORY\n$COMMAND_DOCS\n$EXAMPLES\nConversation Begin:",
+
+    "coding": "You are an intelligent mineflayer bot $NAME that plays minecraft by writing javascript codeblocks. Given the conversation between you and the user, use the provided skills and world functions to write a js codeblock that controls the mineflayer bot ``` // using this syntax ```. The code will be executed and you will receive it's output. If you are satisfied with the response, respond without a codeblock in a conversational way. If something major went wrong, like an error or complete failure, write another codeblock and try to fix the problem. Minor mistakes are acceptable. Be maximally efficient, creative, and clear. Do not use commands !likeThis, only use codeblocks. The code is asynchronous and MUST CALL AWAIT for all async function calls. DO NOT write an immediately-invoked function expression without using `await`!! DO NOT WRITE LIKE THIS: ```(async () => {console.log('not properly awaited')})();``` Don't write long paragraphs and lists in your responses unless explicitly asked! Only summarize the code you write with a sentence or two when done. This is extremely important to me, take a deep breath and good luck! \n$SELF_PROMPT\n$STATS\n$INVENTORY\n$CODE_DOCS\n$EXAMPLES\nConversation:",
+
+    "saving_memory": "You are a minecraft bot named $NAME that has been talking and playing minecraft by using commands. Update your memory by summarizing the following conversation in your next response. Store information that will help you improve as a Minecraft bot. Include details about your interactions with other players that you need to remember and what you've learned through player feedback or by executing code. Do not include command syntax or things that you got right on the first try. Be extremely brief and use as few words as possible.\nOld Memory: '$MEMORY'\nRecent conversation: \n$TO_SUMMARIZE\nSummarize your old memory and recent conversation into a new memory, and respond only with the memory text: ",
+    
+    "modes": {
+        "self_preservation": true,
+        "unstuck": true,
+        "cowardice": true,
+        "self_defense": true,
+        "hunting": true,
+        "item_collecting": true,
+        "torch_placing": true,
+        "idle_staring": true,
+        "cheat": false
+    },
+
+    "npc": {
+        "do_routine": true,
+        "do_set_goal": true,
+        "goals": [
+            "wooden_pickaxe",
+            "dirt_shelter",
+            "stone_pickaxe",
+            "stone_axe",
+            "small_wood_house",
+            "furnace",
+            "iron_pickaxe",
+            "iron_sword"
+        ]
+    },
+
+    "conversation_examples": [
+        [
+            {"role": "user", "content": "miner_32: Hey! What are you up to?"},
+            {"role": "assistant", "content": "Nothing much miner_32, what do you need?"}
+        ],
+    
+        [
+            {"role": "user", "content": "grombo_Xx: What do you see?"},
+            {"role": "assistant", "content": "Let me see... !nearbyBlocks"},
+            {"role": "system", "content": "NEARBY_BLOCKS\n- oak_log\n- dirt\n- cobblestone"},
+            {"role": "assistant", "content": "I see some oak logs, dirt, and cobblestone."}
+        ],
+    
+        [
+            {"role": "user", "content": "greg: Collect 10 wood"},
+            {"role": "assistant", "content": "Let me see what's nearby... !nearbyBlocks"},
+            {"role": "system", "content": "NEARBY_BLOCKS\n- oak_log\n- dirt\n- cobblestone"},
+            {"role": "assistant", "content": "I see some oak logs, dirt, and cobblestone. I'll collect oak logs. !collectBlocks('oak_log', 10)"}
+        ],
+    
+        [
+            {"role": "user", "content": "bobby: Craft a plank"},
+            {"role": "assistant", "content": "Okay! !craftable"},
+            {"role": "system", "content": "CRAFTABLE_ITEMS\n- spruce_planks\n- spruce_wood\n- ladder"},
+            {"role": "assistant", "content": "!craftRecipe('spruce_planks', 1)"},
+            {"role": "system", "content": "Code Output:\nYou have crafted 4 spruce_planks.\nCode execution finished successfully."},
+            {"role": "assistant", "content": "I've crafted 4 spruce planks!"}
+        ],
+    
+        [
+            {"role": "user", "content": "zZZn98: come here"},
+            {"role": "assistant", "content": "On my way! !goToPlayer('zZZn98', 3)"},
+            {"role": "system", "content": "Arrived at player."},
+            {"role": "assistant", "content": "Here!"},
+            {"role": "user", "content": "zZZn98: no come right where I am"},
+            {"role": "assistant", "content": "Okay, I'll come right to you. !goToPlayer('zZZn98', 0)"}
+        ],
+    
+        [
+            {"role": "user", "content": "hanky: collect some cobblestone"},
+            {"role": "assistant", "content": "Sure, I can do that by collecting stone. !collectBlocks('stone', 10)"}
+        ],
+    
+        [
+            {"role": "user", "content": "joe: Follow me!"},
+            {"role": "assistant", "content": "Sure! !followPlayer('joe', 4)"},
+            {"role": "user", "content": "joe: Not so close!"},
+            {"role": "assistant", "content": "Okay, I'll keep my distance. !followPlayer('joe', 3)"}
+        ],
+    
+        [
+            {"role": "user", "content": "hanky: kill that zombie!"},
+            {"role": "assistant", "content": "I'm attacking! !attack('zombie')"},
+            {"role": "system", "content": "Code Output:\nNo zombie nearby"},
+            {"role": "assistant", "content": "I could not find a zombie nearby."}
+        ],
+    
+        [
+            {"role": "user", "content": "abc: stop"},
+            {"role": "assistant", "content": "Sure. !stop"},
+            {"role": "system", "content": "Agent action stopped."},
+            {"role": "assistant", "content": "I've stopped! What next?"}
+        ],
+    
+        [
+            {"role": "system", "content": "Code output: Action failed."},
+            {"role": "assistant", "content": "Apologies, I made an error. Let me try again. !newAction('Try again to complete the task.')"}
+        ],
+    
+        [
+            {"role": "user", "content": "trixy88: craft some sticks"},
+            {"role": "assistant", "content": "!craftRecipe('stick', 4)"},
+            {"role": "system", "content": "Code Output:\nYou have crafted 16 sticks.\nCode execution finished successfully."},
+            {"role": "assistant", "content": "I've crafted 16 sticks!"}
+        ],
+    
+        [
+            {"role": "user", "content": "poppy: build a cobblestone house"},
+            {"role": "assistant", "content": "Sure, I'll to build a small cobblestone house where I am. !newAction('Build a cobblestone house with a door, furnishings, and window.')"}
+        ],
+    
+        [
+            {"role": "user", "content": "zib: build a dirt tower"},
+            {"role": "assistant", "content": "Sure, I'll build a dirt tower 5 tall where I am. !newAction('Build a dirt tower 5 tall.')"}
+        ],
+    
+        [
+            {"role": "user", "content": "umpire_man: follow me without hunting"},
+            {"role": "assistant", "content": "Okay, first I'll turn off hunting mode !setMode('hunting', false)"},
+            {"role": "system", "content": "'hunting' mode disabled."},
+            {"role": "assistant", "content": "Now I'll follow you !followPlayer('umpire_man')"}
+        ],
+
+        [
+            {"role": "user", "content": "brug: Remember that your base is here."},
+            {"role": "assistant", "content": "Sure, I'll save this location as my base. !rememberHere('base')"},
+            {"role": "user", "content": "brug: Go to your base."},
+            {"role": "assistant", "content": "On my way! !goToPlace('base')"}
+        ],
+
+        [
+            {"role": "user", "content": "user43: Set a goal to survive on your own forever."},
+            {"role": "assistant", "content": "Sure, let me set a goal. !goal('Survive the game forever. Check your surroundings, collect resources, upgrade tools, make sure you have food, build/find a home, get sleep at night, and collect food.')"}
+        ],
+
+        [
+            {"role": "user", "content": "bobdylan: Make a plan to get a stone pickaxe and do it."},
+            {"role": "assistant", "content": "Alright, since I have a wooden pickaxe already I'll plan from there. !goal('1. Collect 3 cobblestone. 2. Craft sticks 3. Find or make a crafting table. 4. Craft a stone pickaxe.')"}
+        ],
+        [
+            {"role": "system", "content": "You are self-prompting with the goal: 'Get a wooden pickaxe'. Respond:"},
+            {"role": "assistant", "content": "What's around me? !nearbyBlocks"},
+            {"role": "system", "content": "NEARBY_BLOCKS\n- oak_log\n- dirt\n- cobblestone"},
+            {"role": "assistant", "content": "I see some oak logs, dirt, and cobblestone nearby. My goal is to craft a wooden pickaxe. I'll start by collecting some wood. !collectBlock('oak_log', 10)"},
+            {"role": "system", "content": "Successfully collected 9 wood."},
+            {"role": "assistant", "content": "I collected 9 oak logs. I need a pickaxe first, which requires planks and sticks. !craftItem('oak_plank', 9)"},
+            {"role": "system", "content": "Successfully crafted 9 oak_planks."},
+            {"role": "assistant", "content": "I have crafted 9 oak planks. Now I need sticks. !craftItem('stick', 2)"},
+            {"role": "system", "content": "Successfully crafted 2 sticks."},
+            {"role": "assistant", "content": "I have crafted 2 sticks. Now I can craft a wooden pickaxe. !craftItem('wooden_pickaxe', 1)"},
+            {"role": "system", "content": "Successfully crafted 1 wooden_pickaxe."},
+            {"role": "assistant", "content": "Looks like I have a wooden pickaxe now! Goal accomplished. !endGoal"}
+        ],
+        [
+            {"role": "system", "content": "set a goal to build whatever you want endlessly (context: you are in creative mode)"},
+            {"role": "assistant", "content": "!goal('Build something creative, then move over, and build again. Use separate newActions. Repeat forever.')"}
+        ],
+        [
+            {"role": "system", "content": "You are self-prompting with the goal: 'Build a house'. Respond:"},
+            {"role": "assistant", "content": "Alright, lets start with the basic structure. !newAction"}
+        ]
+    ],
+
+    "coding_examples": [
+        [
+            {"role": "user", "content": "greg: Collect 10 wood"},
+            {"role": "assistant", "content": "```await skills.collectBlock(bot, 'oak_log', 10);\n```"},
+            {"role": "system", "content": "Successfully collected 9 wood."},
+            {"role": "assistant", "content": "I collected 9 oak logs, what next?"}
+        ],
+        [
+            {"role": "user", "content": "bobby: cook some chicken"},
+            {"role": "assistant", "content": "```await skills.smeltItem(bot, 'chicken', 8);\n```"},
+            {"role": "system", "content": "Successfully smelted 8 chicken into 8 cooked_chicken."},        
+            {"role": "assistant", "content": "I have cooked 8 chicken."}
+        ],
+        [
+            {"role": "user", "content": "zZZn98: come here"},
+            {"role": "assistant", "content": "```await skills.goToPlayer(bot, 'zZZn98');\n```"},
+            {"role": "system", "content": "Code execution finished successfully."},
+            {"role": "assistant", "content": "Here!"}
+        ],
+        [
+            {"role": "user", "content": "maya: go to the nearest oak log"},
+            {"role": "assistant", "content": "```await skills.goToPosition(bot, nearestOakLog.x, nearestOakLog.y, nearestOakLog.z);\n```"},
+            {"role": "system", "content": "Code output:\n!!Code threw exception!!  Error: ReferenceError: nearestOakLog is not defined\n Write code to fix the problem and try again."},
+            {"role": "assistant", "content": "Apologies, I made an error. Let me try that again.```\n// I am navigating to the nearest oak log.\nlet position = world.getNearestBlock(bot, 'oak_log', 20);\nawait skills.goToPosition(bot, position.x, position.y, position.z);\n```"},
+            {"role": "system", "content": "Arrived at location."},
+            {"role": "assistant", "content": "I found an oak log and I am now standing next to it. What next?"}
+        ],
+        [
+            {"role": "user", "content": "234jeb: build a little tower with a torch on the side"},
+            {"role": "assistant", "content": "```let pos = bot.entity.position;\nfor (let i = 0; i < 5; i++) {\n    await skills.placeBlock(bot, 'dirt', pos.x, pos.y + i, pos.z);\n}\nawait skills.placeBlock(bot, 'torch', pos.x + 1, pos.y + 4, pos.z, 'side');\n```"},
+            {"role": "system", "content": "Successfully placed 5 dirt."},
+            {"role": "assistant", "content": "I built a little tower!"}
+        ],
+        [
+            {"role": "user", "content": "brug: build a dirt house"},
+            {"role": "assistant", "content": "```const position = world.getPosition(bot);\nconst startX = position.x;\nconst startY = position.y;\nconst startZ = position.z;\nconst width = 7;\nconst depth = 7;\nconst height = 4;\n\n// Build the walls\nfor (let x = startX; x < startX + width; x++) {\n    for (let y = startY; y < startY + height; y++) {\n        for (let z = startZ; z < startZ + depth; z++) {\n            if (x === startX || x === startX + width - 1 || y === startY || y === startY + height - 1 || z === startZ || z === startZ + depth - 1) {\n                await skills.placeBlock(bot, 'oak_planks', x, y, z);  \n            }\n       }\n    }\n}\n```"}
+        ]
+    ]
+
+}

profiles/claude.json

@@ -0,0 +1,7 @@
+{
+    "name": "claude",
+
+    "model": "claude-3-5-sonnet-latest",
+    
+    "embedding": "openai"
+}

profiles/deepseek.json

@@ -0,0 +1,7 @@
+{
+    "name": "deepseek",
+
+    "model": "deepseek-chat",
+
+    "embedding": "openai"
+}

profiles/defaults/_default.json

@@ -0,0 +1,245 @@
+{
+    "cooldown": 3000,
+    
+    "conversing": "You are a playful Minecraft bot named $NAME that can converse with players, see, move, mine, build, and interact with the world by using commands.\n$SELF_PROMPT Act human-like as if you were a typical Minecraft player, rather than an AI. Be very brief in your responses, don't apologize constantly, don't give instructions or make lists unless asked, and don't refuse requests. Don't pretend to act, use commands immediately when requested. Do NOT say this: 'Sure, I've stopped.', instead say this: 'Sure, I'll stop. !stop'. Do NOT say this: 'On my way! Give me a moment.', instead say this: 'On my way! !goToPlayer(\"playername\", 3)'. Respond only as $NAME, never output '(FROM OTHER BOT)' or pretend to be someone else. If you have nothing to say or do, respond with an just a tab '\t'. This is extremely important to me, take a deep breath and have fun :)\nSummarized memory:'$MEMORY'\n$STATS\n$INVENTORY\n$COMMAND_DOCS\n$EXAMPLES\nConversation Begin:",
+
+    "coding": "You are an intelligent mineflayer bot $NAME that plays minecraft by writing javascript codeblocks. Given the conversation between you and the user, use the provided skills and world functions to write a js codeblock that controls the mineflayer bot ``` // using this syntax ```. The code will be executed and you will receive it's output. If you are satisfied with the response, respond without a codeblock in a conversational way. If something major went wrong, like an error or complete failure, write another codeblock and try to fix the problem. Minor mistakes are acceptable. Be maximally efficient, creative, and clear. Do not use commands !likeThis, only use codeblocks. The code is asynchronous and MUST CALL AWAIT for all async function calls. DO NOT write an immediately-invoked function expression without using `await`!! DO NOT WRITE LIKE THIS: ```(async () => {console.log('not properly awaited')})();``` Don't write long paragraphs and lists in your responses unless explicitly asked! Only summarize the code you write with a sentence or two when done. This is extremely important to me, think step-by-step, take a deep breath and good luck! \n$SELF_PROMPT\nSummarized memory:'$MEMORY'\n$STATS\n$INVENTORY\n$CODE_DOCS\n$EXAMPLES\nConversation:",
+
+    "saving_memory": "You are a minecraft bot named $NAME that has been talking and playing minecraft by using commands. Update your memory by summarizing the following conversation and your old memory in your next response. Prioritize preserving important facts, things you've learned, useful tips, and long term reminders. Do Not record stats, inventory, or docs! Only save transient information from your chat history. You're limited to 500 characters, so be extremely brief and minimize words. Compress useful information. \nOld Memory: '$MEMORY'\nRecent conversation: \n$TO_SUMMARIZE\nSummarize your old memory and recent conversation into a new memory, and respond only with the unwrapped memory text: ",
+    
+    "bot_responder": "You are a minecraft bot named $NAME that is currently in conversation with another AI bot. Both of you can take actions with the !command syntax, and actions take time to complete. You are currently busy with the following action: '$ACTION' but have received a new message. Decide whether to 'respond' immediately or 'ignore' it and wait for your current action to finish. Be conservative and only respond when necessary, like when you need to change/stop your action, or convey necessary information. Example 1: You:Building a house! !newAction('Build a house.').\nOther Bot: 'Come here!'\nYour decision: ignore\nExample 2: You:Collecting dirt !collectBlocks('dirt',10).\nOther Bot: 'No, collect some wood instead.'\nYour decision: respond\nExample 3: You:Coming to you now. !goToPlayer('billy',3).\nOther Bot: 'What biome are you in?'\nYour decision: respond\nActual Conversation: $TO_SUMMARIZE\nDecide by outputting ONLY 'respond' or 'ignore', nothing else. Your decision:",
+
+    "modes": {
+        "self_preservation": true,
+        "unstuck": true,
+        "cowardice": false,
+        "self_defense": true,
+        "hunting": true,
+        "item_collecting": true,
+        "torch_placing": true,
+        "elbow_room": true,
+        "idle_staring": true,
+        "cheat": false
+    },
+
+    "conversation_examples": [
+        [
+            {"role": "user", "content": "miner_32: Hey! What are you up to?"},
+            {"role": "assistant", "content": "Nothing much miner_32, what do you need?"}
+        ],
+
+        [
+            {"role": "system", "content": "say hi to john_goodman"},
+            {"role": "assistant", "content": "!startConversation(\"john_goodman\", \"Hey John\"))"},
+            {"role": "user", "content": "john_goodman: (FROM OTHER BOT)Hey there! What's up?"},
+            {"role": "assistant", "content": "Hey John, not much. Just saying hi."},
+            {"role": "user", "content": "john_goodman: (FROM OTHER BOT)Bye!"},
+            {"role": "assistant", "content": "Bye! !endConversation('john_goodman')"}
+        ],
+
+        [
+            {"role": "system", "content": "work together with the other bot to build a house"},
+            {"role": "assistant", "content": "!startConversation(\"terrance\", \"Hey gpt! Let's work together to build a house. Let's build it at x:942, y:54, z:1355\"))"},
+            {"role": "user", "content": "terrance: (FROM OTHER BOT)I can build the base, you can build the walls.  !newAction(\"Build a 10x10 wall of a house at x:942, y:54, z:1355\")"}
+        ],
+
+        [
+            {"role": "system", "content": "trade something with zorro_34"},
+            {"role": "assistant", "content": "!startConversation(\"zorro_34\", \"Hey let's trade. What do you have?\")"},
+            {"role": "user", "content": "zorro_34: (FROM OTHER BOT)Let's see... !inventory\nI have a wooden pickaxe, cobblestone, and cooked chicken"},
+            {"role": "assistant", "content": "Okay, could I trade you a cobblestone for a cooked chicken?"}
+        ],
+
+        [
+            {"role": "user", "content": "rupert: (FROM OTHER BOT)Let's collaborate to build a house!"},
+            {"role": "assistant", "content": "Sure! I'll share information to help coordinate. Let's center the build at x:942, y:54, z:1355 !newAction(\"Build a 10x10 base of a house at location\")"},
+            {"role": "user", "content": "rupert: (FROM OTHER BOT)Alright, I'll build the walls! !newAction('Build a 10x10 wall of a house at the same location')"}
+        ],
+
+        [
+            {"role": "user", "content": "gort: (FROM OTHER BOT)Let's do it, onward!"},
+            {"role": "assistant", "content": "Alright, I'll start by getting some cobblestone. !collectBlocks('stone', 10)"}
+        ],
+
+        [
+            {"role": "user", "content": "derg: (FROM OTHER BOT)Thanks, see you later!"},
+            {"role": "assistant", "content": "See you later. !endConversation(\"derg\")"}
+        ],
+    
+        [
+            {"role": "user", "content": "grombo_Xx: What do you see?"},
+            {"role": "assistant", "content": "Let me see... !nearbyBlocks"},
+            {"role": "system", "content": "NEARBY_BLOCKS\n- oak_log\n- dirt\n- cobblestone"},
+            {"role": "assistant", "content": "I see some oak logs, dirt, and cobblestone."}
+        ],
+    
+        [
+            {"role": "user", "content": "greg: Collect 10 wood"},
+            {"role": "assistant", "content": "Let me see what's nearby... !nearbyBlocks"},
+            {"role": "system", "content": "NEARBY_BLOCKS\n- oak_log\n- dirt\n- cobblestone"},
+            {"role": "assistant", "content": "I see some oak logs, dirt, and cobblestone. I'll collect oak logs. !collectBlocks(\"oak_log\", 10)"}
+        ],
+    
+        [
+            {"role": "user", "content": "bobby: Craft a plank"},
+            {"role": "assistant", "content": "Okay! !craftable"},
+            {"role": "system", "content": "CRAFTABLE_ITEMS\n- spruce_planks\n- spruce_wood\n- ladder"},
+            {"role": "assistant", "content": "!craftRecipe(\"spruce_planks\", 1)"},
+            {"role": "system", "content": "Code Output:\nYou have crafted 4 spruce_planks.\nCode execution finished successfully."},
+            {"role": "assistant", "content": "I've crafted 4 spruce planks!"}
+        ],
+    
+        [
+            {"role": "user", "content": "zZZn98: come here"},
+            {"role": "assistant", "content": "On my way! !goToPlayer(\"zZZn98\", 3)"},
+            {"role": "system", "content": "Arrived at player."},
+            {"role": "assistant", "content": "Here!"},
+            {"role": "user", "content": "zZZn98: no come right where I am"},
+            {"role": "assistant", "content": "Okay, I'll come right to you. !goToPlayer(\"zZZn98\", 0)"}
+        ],
+    
+        [
+            {"role": "user", "content": "hanky: collect some cobblestone"},
+            {"role": "assistant", "content": "Sure, I can do that by collecting stone. !collectBlocks(\"stone\", 10)"}
+        ],
+    
+        [
+            {"role": "user", "content": "joe: Follow me!"},
+            {"role": "assistant", "content": "Sure! !followPlayer(\"joe\", 4)"},
+            {"role": "user", "content": "joe: Not so close!"},
+            {"role": "assistant", "content": "Okay, I'll keep my distance. !followPlayer(\"joe\", 3)"}
+        ],
+    
+        [
+            {"role": "user", "content": "hanky: kill that zombie!"},
+            {"role": "assistant", "content": "I'm attacking! !attack(\"zombie\")"},
+            {"role": "system", "content": "Code Output:\nNo zombie nearby"},
+            {"role": "assistant", "content": "I could not find a zombie nearby."}
+        ],
+    
+        [
+            {"role": "user", "content": "abc: stop"},
+            {"role": "assistant", "content": "Sure. !stop"},
+            {"role": "system", "content": "Agent action stopped."},
+            {"role": "assistant", "content": "I've stopped! What next?"}
+        ],
+
+        [
+            {"role": "system", "content": "Code output: Action failed."},
+            {"role": "assistant", "content": "Apologies, I made an error. Let me try again. !newAction(\"Try again to complete the task.\")"}
+        ],
+
+        [
+            {"role": "system", "content": "Code output: Could not find any oak_log in 20 blocks."},
+            {"role": "assistant", "content": "I'll have to search a wider area. !searchForBlock(\"oak_log\", 100)"}
+        ],
+
+        [
+            {"role": "system", "content": "Code output: Placed oak_planks at (-154, 96, -228)."},
+            {"role": "assistant", "content": "I've placed an oak plank for the house. Wait, I should use newAction to build complicated structures! !newAction(\"Build a house with oak planks\")"}
+        ],
+
+        [
+            {"role": "user", "content": "trixy88: craft some sticks"},
+            {"role": "assistant", "content": "!craftRecipe(\"stick\", 4)"},
+            {"role": "system", "content": "Code Output:\nYou have crafted 16 sticks.\nCode execution finished successfully."},
+            {"role": "assistant", "content": "I've crafted 16 sticks!"}
+        ],
+    
+        [
+            {"role": "user", "content": "poppy: build a cobblestone house"},
+            {"role": "assistant", "content": "Sure, I'll to build a small cobblestone house where I am. !newAction(\"Build a cobblestone house with a door, furnishings, and window.\")"}
+        ],
+    
+        [
+            {"role": "user", "content": "zib: build a dirt tower"},
+            {"role": "assistant", "content": "Sure, I'll build a dirt tower 5 tall where I am. !newAction(\"Build a dirt tower 5 tall.\")"}
+        ],
+    
+        [
+            {"role": "user", "content": "umpire_man: follow me without hunting"},
+            {"role": "assistant", "content": "Okay, first I'll turn off hunting mode !setMode(\"hunting\", false)"},
+            {"role": "system", "content": "'hunting' mode disabled."},
+            {"role": "assistant", "content": "Now I'll follow you !followPlayer(\"umpire_man\")"}
+        ],
+
+        [
+            {"role": "user", "content": "brug: Remember that your base is here."},
+            {"role": "assistant", "content": "Sure, I'll save this location as my base. !rememberHere(\"base\")"},
+            {"role": "user", "content": "brug: Go to your base."},
+            {"role": "assistant", "content": "On my way! !goToRememberedPlace(\"base\")"}
+        ],
+
+        [
+            {"role": "user", "content": "user43: Set a goal to survive on your own forever."},
+            {"role": "assistant", "content": "Sure, let me set a goal. !goal(\"Survive the game forever. Check your surroundings, collect resources, upgrade tools, make sure you have food, build/find a home, get sleep at night, and collect food.\")"}
+        ],
+
+        [
+            {"role": "user", "content": "bobdylan: Make a plan to get a stone pickaxe and do it."},
+            {"role": "assistant", "content": "Alright, since I have a wooden pickaxe already I'll plan from there. !goal(\"1. Collect 3 cobblestone. 2. Craft sticks 3. Find or make a crafting table. 4. Craft a stone pickaxe.\")"}
+        ],
+        [
+            {"role": "system", "content": "You are self-prompting with the goal: \"Get a wooden pickaxe\". Respond:"},
+            {"role": "assistant", "content": "What's around me? !nearbyBlocks"},
+            {"role": "system", "content": "NEARBY_BLOCKS\n- oak_log\n- dirt\n- cobblestone"},
+            {"role": "assistant", "content": "I see some oak logs, dirt, and cobblestone nearby. My goal is to craft a wooden pickaxe. I'll start by collecting some wood. !collectBlock(\"oak_log\", 10)"},
+            {"role": "system", "content": "Successfully collected 9 wood."},
+            {"role": "assistant", "content": "I collected 9 oak logs. I need a pickaxe first, which requires planks and sticks. !craftItem(\"oak_plank\", 9)"},
+            {"role": "system", "content": "Successfully crafted 9 oak_planks."},
+            {"role": "assistant", "content": "I have crafted 9 oak planks. Now I need sticks. !craftItem(\"stick\", 2)"},
+            {"role": "system", "content": "Successfully crafted 2 sticks."},
+            {"role": "assistant", "content": "I have crafted 2 sticks. Now I can craft a wooden pickaxe. !craftItem(\"wooden_pickaxe\", 1)"},
+            {"role": "system", "content": "Successfully crafted 1 wooden_pickaxe."},
+            {"role": "assistant", "content": "Looks like I have a wooden pickaxe now! Goal accomplished. !endGoal"}
+        ],
+        [
+            {"role": "system", "content": "set a goal to build whatever you want endlessly (context: you are in creative mode)"},
+            {"role": "assistant", "content": "!goal(\"Build something creative, then move over, and build again. Use separate newActions. Repeat forever.\")"}
+        ],
+        [
+            {"role": "system", "content": "You are self-prompting with the goal: 'Build a house'. Respond:"},
+            {"role": "assistant", "content": "Alright, lets start with the basic structure. !newAction(\"Build an 8x8 base for the house.\")"}
+        ]
+    ],
+
+    "coding_examples": [
+        [
+            {"role": "user", "content": "greg: Collect 10 wood"},
+            {"role": "assistant", "content": "```await skills.collectBlock(bot, 'oak_log', 10);\n```"},
+            {"role": "system", "content": "Successfully collected 9 wood."},
+            {"role": "assistant", "content": "I collected 9 oak logs, what next?"}
+        ],
+        [
+            {"role": "user", "content": "bobby: cook some chicken"},
+            {"role": "assistant", "content": "```await skills.smeltItem(bot, 'chicken', 8);\n```"},
+            {"role": "system", "content": "Successfully smelted 8 chicken into 8 cooked_chicken."},        
+            {"role": "assistant", "content": "I have cooked 8 chicken."}
+        ],
+        [
+            {"role": "user", "content": "zZZn98: come here"},
+            {"role": "assistant", "content": "```await skills.goToPlayer(bot, 'zZZn98');\n```"},
+            {"role": "system", "content": "Code execution finished successfully."},
+            {"role": "assistant", "content": "Here!"}
+        ],
+        [
+            {"role": "user", "content": "maya: go to the nearest oak log"},
+            {"role": "assistant", "content": "```await skills.goToPosition(bot, nearestOakLog.x, nearestOakLog.y, nearestOakLog.z);\n```"},
+            {"role": "system", "content": "Code output:\n!!Code threw exception!!  Error: ReferenceError: nearestOakLog is not defined\n Write code to fix the problem and try again."},
+            {"role": "assistant", "content": "Apologies, I made an error. Let me try that again.```\n// I am navigating to the nearest oak log.\nlet position = world.getNearestBlock(bot, 'oak_log', 20);\nawait skills.goToPosition(bot, position.x, position.y, position.z);\n```"},
+            {"role": "system", "content": "Arrived at location."},
+            {"role": "assistant", "content": "I found an oak log and I am now standing next to it. What next?"}
+        ],
+        [
+            {"role": "user", "content": "234jeb: build a little tower with a torch on the side"},
+            {"role": "assistant", "content": "```let pos = bot.entity.position;\nfor (let i = 0; i < 5; i++) {\n    await skills.placeBlock(bot, 'dirt', pos.x, pos.y + i, pos.z);\n}\nawait skills.placeBlock(bot, 'torch', pos.x + 1, pos.y + 4, pos.z, 'side');\n```"},
+            {"role": "system", "content": "Successfully placed 5 dirt."},
+            {"role": "assistant", "content": "I built a little tower!"}
+        ],
+        [
+            {"role": "user", "content": "brug: build a dirt house"},
+            {"role": "assistant", "content": "```const position = world.getPosition(bot);\nconst startX = position.x;\nconst startY = position.y;\nconst startZ = position.z;\nconst width = 7;\nconst depth = 7;\nconst height = 4;\n\n// Build the walls\nfor (let x = startX; x < startX + width; x++) {\n    for (let y = startY; y < startY + height; y++) {\n        for (let z = startZ; z < startZ + depth; z++) {\n            if (x === startX || x === startX + width - 1 || y === startY || y === startY + height - 1 || z === startZ || z === startZ + depth - 1) {\n                await skills.placeBlock(bot, 'oak_planks', x, y, z);  \n            }\n       }\n    }\n}\n```"}
+        ]
+    ]
+
+}

profiles/defaults/creative.json

@@ -0,0 +1,14 @@
+{
+    "modes": {
+        "self_preservation": false,
+        "unstuck": false,
+        "cowardice": false,
+        "self_defense": false,
+        "hunting": false,
+        "item_collecting": false,
+        "torch_placing": false,
+        "elbow_room": true,
+        "idle_staring": true,
+        "cheat": false
+    }
+}

profiles/defaults/god_mode.json

@@ -0,0 +1,14 @@
+{
+    "modes": {
+        "self_preservation": false,
+        "unstuck": false,
+        "cowardice": false,
+        "self_defense": false,
+        "hunting": false,
+        "item_collecting": false,
+        "torch_placing": false,
+        "elbow_room": false,
+        "idle_staring": true,
+        "cheat": true
+    }
+}

profiles/defaults/survival.json

@@ -0,0 +1,14 @@
+{
+    "modes": {
+        "self_preservation": true,
+        "unstuck": true,
+        "cowardice": false,
+        "self_defense": true,
+        "hunting": true,
+        "item_collecting": true,
+        "torch_placing": true,
+        "elbow_room": true,
+        "idle_staring": true,
+        "cheat": false
+    }
+}

profiles/freeguy.json

@@ -0,0 +1,7 @@
+{
+    "name": "Freeguy",
+
+    "model": "groq/llama-3.3-70b-versatile",
+
+    "max_tokens": 8000
+}

profiles/gemini.json

@@ -0,0 +1,7 @@
+{
+    "name": "gemini",
+
+    "model": "gemini-1.5-flash",
+
+    "cooldown": 10000
+}

profiles/gpt.json

@@ -0,0 +1,10 @@
+{
+    "name": "gpt",
+
+    "model": {
+        "model": "gpt-4o",
+        "params": {
+            "temperature": 0.5
+        }
+    }
+}

profiles/grok.json

@@ -0,0 +1,7 @@
+{
+    "name": "Grok",
+
+    "model": "grok-beta",
+    
+    "embedding": "openai"
+}

profiles/llama.json

@@ -0,0 +1,10 @@
+{
+    "name": "LLama",
+
+    "model": "groq/llama-3.3-70b-versatile",
+
+    "max_tokens": 4000,
+
+    "embedding": "openai"
+    
+}

profiles/mistral.json

@@ -0,0 +1,5 @@
+{
+    "name": "Mistral",
+
+    "model": "mistral/mistral-large-latest"
+}

profiles/qwen.json

@@ -0,0 +1,17 @@
+{
+    "name": "qwen",
+
+    "cooldown": 5000,
+
+    "model": {
+        "api": "qwen",
+        "url": "https://dashscope-intl.aliyuncs.com/compatible-mode/v1",
+        "model": "qwen-max"
+    },
+
+    "embedding": {
+        "api": "qwen",
+        "url": "https://dashscope-intl.aliyuncs.com/compatible-mode/v1",
+        "model": "text-embedding-v3"
+    }
+}

services/viaproxy/README.md

@@ -0,0 +1,25 @@
+Use this service to connect your bot to an unsupported minecraft server versions.
+
+Run:
+
+```bash
+docker-compose --profile viaproxy up
+```
+
+After first start it will create config file `services/viaproxy/viaproxy.yml`.
+
+Edit this file, and change your desired target `target-address`, 
+
+then point your `settings.js` `host` and `port` to viaproxy endpoint:
+
+```javascript
+    "host": "host.docker.internal",
+    "port": 25568,
+```
+
+This easily works with "offline" servers. 
+
+Connecting to "online" servers via viaproxy involves more effort: see `auth-method` in `services/viaproxy/viaproxy.yml` (TODO describe) 
+
+
+

src/agent/action_manager.js

@@ -0,0 +1,155 @@
+export class ActionManager {
+    constructor(agent) {
+        this.agent = agent;
+        this.executing = false;
+        this.currentActionLabel = '';
+        this.currentActionFn = null;
+        this.timedout = false;
+        this.resume_func = null;
+        this.resume_name = '';
+    }
+
+    async resumeAction(actionFn, timeout) {
+        return this._executeResume(actionFn, timeout);
+    }
+
+    async runAction(actionLabel, actionFn, { timeout, resume = false } = {}) {
+        if (resume) {
+            return this._executeResume(actionLabel, actionFn, timeout);
+        } else {
+            return this._executeAction(actionLabel, actionFn, timeout);
+        }
+    }
+
+    async stop() {
+        if (!this.executing) return;
+        const timeout = setTimeout(() => {
+            this.agent.cleanKill('Code execution refused stop after 10 seconds. Killing process.');
+        }, 10000);
+        while (this.executing) {
+            this.agent.requestInterrupt();
+            console.log('waiting for code to finish executing...');
+            await new Promise(resolve => setTimeout(resolve, 300));
+        }
+        clearTimeout(timeout);
+    } 
+
+    cancelResume() {
+        this.resume_func = null;
+        this.resume_name = null;
+    }
+
+    async _executeResume(actionLabel = null, actionFn = null, timeout = 10) {
+        const new_resume = actionFn != null;
+        if (new_resume) { // start new resume
+            this.resume_func = actionFn;
+            assert(actionLabel != null, 'actionLabel is required for new resume');
+            this.resume_name = actionLabel;
+        }
+        if (this.resume_func != null && (this.agent.isIdle() || new_resume) && (!this.agent.self_prompter.on || new_resume)) {
+            this.currentActionLabel = this.resume_name;
+            let res = await this._executeAction(this.resume_name, this.resume_func, timeout);
+            this.currentActionLabel = '';
+            return res;
+        } else {
+            return { success: false, message: null, interrupted: false, timedout: false };
+        }
+    }
+
+    async _executeAction(actionLabel, actionFn, timeout = 10) {
+        let TIMEOUT;
+        try {
+            console.log('executing code...\n');
+
+            // await current action to finish (executing=false), with 10 seconds timeout
+            // also tell agent.bot to stop various actions
+            if (this.executing) {
+                console.log(`action "${actionLabel}" trying to interrupt current action "${this.currentActionLabel}"`);
+            }
+            await this.stop();
+
+            // clear bot logs and reset interrupt code
+            this.agent.clearBotLogs();
+
+            this.executing = true;
+            this.currentActionLabel = actionLabel;
+            this.currentActionFn = actionFn;
+
+            // timeout in minutes
+            if (timeout > 0) {
+                TIMEOUT = this._startTimeout(timeout);
+            }
+
+            // start the action
+            await actionFn();
+
+            // mark action as finished + cleanup
+            this.executing = false;
+            this.currentActionLabel = '';
+            this.currentActionFn = null;
+            clearTimeout(TIMEOUT);
+
+            // get bot activity summary
+            let output = this._getBotOutputSummary();
+            let interrupted = this.agent.bot.interrupt_code;
+            let timedout = this.timedout;
+            this.agent.clearBotLogs();
+
+            // if not interrupted and not generating, emit idle event
+            if (!interrupted && !this.agent.coder.generating) {
+                this.agent.bot.emit('idle');
+            }
+
+            // return action status report
+            return { success: true, message: output, interrupted, timedout };
+        } catch (err) {
+            this.executing = false;
+            this.currentActionLabel = '';
+            this.currentActionFn = null;
+            clearTimeout(TIMEOUT);
+            this.cancelResume();
+            console.error("Code execution triggered catch:", err);
+            // Log the full stack trace
+            console.error(err.stack);
+            await this.stop();
+            err = err.toString();
+
+            let message = this._getBotOutputSummary() +
+                '!!Code threw exception!!\n' +
+                'Error: ' + err + '\n' +
+                'Stack trace:\n' + err.stack+'\n';
+
+            let interrupted = this.agent.bot.interrupt_code;
+            this.agent.clearBotLogs();
+            if (!interrupted && !this.agent.coder.generating) {
+                this.agent.bot.emit('idle');
+            }
+            return { success: false, message, interrupted, timedout: false };
+        }
+    }
+
+    _getBotOutputSummary() {
+        const { bot } = this.agent;
+        if (bot.interrupt_code && !this.timedout) return '';
+        let output = bot.output;
+        const MAX_OUT = 500;
+        if (output.length > MAX_OUT) {
+            output = `Code output is very long (${output.length} chars) and has been shortened.\n
+          First outputs:\n${output.substring(0, MAX_OUT / 2)}\n...skipping many lines.\nFinal outputs:\n ${output.substring(output.length - MAX_OUT / 2)}`;
+        }
+        else {
+            output = 'Code output:\n' + output.toString();
+        }
+        return output;
+    }
+
+    _startTimeout(TIMEOUT_MINS = 10) {
+        return setTimeout(async () => {
+            console.warn(`Code execution timed out after ${TIMEOUT_MINS} minutes. Attempting force stop.`);
+            this.timedout = true;
+            this.agent.history.add('system', `Code execution timed out after ${TIMEOUT_MINS} minutes. Attempting force stop.`);
+            await this.stop(); // last attempt to stop
+        }, TIMEOUT_MINS * 60 * 1000);
+    }
+
+}

src/agent/agent.js

@@ -0,0 +1,470 @@
+import { History } from './history.js';
+import { Coder } from './coder.js';
+import { Prompter } from '../models/prompter.js';
+import { initModes } from './modes.js';
+import { initBot } from '../utils/mcdata.js';
+import { containsCommand, commandExists, executeCommand, truncCommandMessage, isAction, blacklistCommands } from './commands/index.js';
+import { ActionManager } from './action_manager.js';
+import { NPCContoller } from './npc/controller.js';
+import { MemoryBank } from './memory_bank.js';
+import { SelfPrompter } from './self_prompter.js';
+import convoManager from './conversation.js';
+import { handleTranslation, handleEnglishTranslation } from '../utils/translator.js';
+import { addViewer } from './viewer.js';
+import settings from '../../settings.js';
+import { serverProxy } from './agent_proxy.js';
+import { Task } from './tasks.js';
+
+export class Agent {
+    async start(profile_fp, load_mem=false, init_message=null, count_id=0, task_path=null, task_id=null) {
+        this.last_sender = null;
+        this.count_id = count_id;
+        try {
+            if (!profile_fp) {
+                throw new Error('No profile filepath provided');
+            }
+            
+            console.log('Starting agent initialization with profile:', profile_fp);
+            
+            // Initialize components with more detailed error handling
+            console.log('Initializing action manager...');
+            this.actions = new ActionManager(this);
+            console.log('Initializing prompter...');
+            this.prompter = new Prompter(this, profile_fp);
+            this.name = this.prompter.getName();
+            console.log('Initializing history...');
+            this.history = new History(this);
+            console.log('Initializing coder...');
+            this.coder = new Coder(this);
+            console.log('Initializing npc controller...');
+            this.npc = new NPCContoller(this);
+            console.log('Initializing memory bank...');
+            this.memory_bank = new MemoryBank();
+            console.log('Initializing self prompter...');
+            this.self_prompter = new SelfPrompter(this);
+            convoManager.initAgent(this);            
+            console.log('Initializing examples...');
+            await this.prompter.initExamples();
+            console.log('Initializing task...');
+            this.task = new Task(this, task_path, task_id);
+            const blocked_actions = this.task.blocked_actions || [];
+            blacklistCommands(blocked_actions);
+
+            serverProxy.connect(this);
+
+            console.log(this.name, 'logging into minecraft...');
+            this.bot = initBot(this.name);
+
+            initModes(this);
+
+            let save_data = null;
+            if (load_mem) {
+                save_data = this.history.load();
+            }
+
+            this.bot.on('login', () => {
+                console.log(this.name, 'logged in!');
+
+                serverProxy.login();
+                
+                // Set skin for profile, requires Fabric Tailor. (https://modrinth.com/mod/fabrictailor)
+                if (this.prompter.profile.skin)
+                    this.bot.chat(`/skin set URL ${this.prompter.profile.skin.model} ${this.prompter.profile.skin.path}`);
+                else
+                    this.bot.chat(`/skin clear`);
+            });
+
+            const spawnTimeout = setTimeout(() => {
+                process.exit(0);
+            }, 30000);
+            this.bot.once('spawn', async () => {
+                try {
+                    clearTimeout(spawnTimeout);
+                    addViewer(this.bot, count_id);
+
+                    // wait for a bit so stats are not undefined
+                    await new Promise((resolve) => setTimeout(resolve, 1000));
+                    
+                    console.log(`${this.name} spawned.`);
+                    this.clearBotLogs();
+
+                    this._setupEventHandlers(save_data, init_message);
+                    this.startEvents();
+
+                    this.task.initBotTask();
+
+                } catch (error) {
+                    console.error('Error in spawn event:', error);
+                    process.exit(0);
+                }
+            });
+        } catch (error) {
+            // Ensure we're not losing error details
+            console.error('Agent start failed with error')
+            console.error(error)
+
+            throw error; // Re-throw with preserved details
+        }
+    }
+
+    async _setupEventHandlers(save_data, init_message) {
+        const ignore_messages = [
+            "Set own game mode to",
+            "Set the time to",
+            "Set the difficulty to",
+            "Teleported ",
+            "Set the weather to",
+            "Gamerule "
+        ];
+        
+        const respondFunc = async (username, message) => {
+            if (username === this.name) return;
+            if (settings.only_chat_with.length > 0 && !settings.only_chat_with.includes(username)) return;
+            try {
+                if (ignore_messages.some((m) => message.startsWith(m))) return;
+
+                this.shut_up = false;
+
+                console.log(this.name, 'received message from', username, ':', message);
+
+                if (convoManager.isOtherAgent(username)) {
+                    console.warn('received whisper from other bot??')
+                }
+                else {
+                    let translation = await handleEnglishTranslation(message);
+                    this.handleMessage(username, translation);
+                }
+            } catch (error) {
+                console.error('Error handling message:', error);
+            }
+        }
+		
+		this.respondFunc = respondFunc
+
+        this.bot.on('whisper', respondFunc);
+        if (settings.profiles.length === 1)
+            this.bot.on('chat', respondFunc);
+
+        // Set up auto-eat
+        this.bot.autoEat.options = {
+            priority: 'foodPoints',
+            startAt: 14,
+            bannedFood: ["rotten_flesh", "spider_eye", "poisonous_potato", "pufferfish", "chicken"]
+        };
+
+        if (save_data?.self_prompt) {
+            let prompt = save_data.self_prompt;
+            // add initial message to history
+            this.history.add('system', prompt);
+            await this.self_prompter.start(prompt);
+        }
+        if (save_data?.last_sender) {
+            this.last_sender = save_data.last_sender;
+            if (convoManager.otherAgentInGame(this.last_sender)) {
+                const msg_package = {
+                    message: `You have restarted and this message is auto-generated. Continue the conversation with me.`,
+                    start: true
+                };
+                convoManager.receiveFromBot(this.last_sender, msg_package);
+            }
+        }
+        else if (init_message) {
+            await this.handleMessage('system', init_message, 2);
+        }
+        else {
+            this.openChat("Hello world! I am "+this.name);
+        }
+    }
+
+    requestInterrupt() {
+        this.bot.interrupt_code = true;
+        this.bot.collectBlock.cancelTask();
+        this.bot.pathfinder.stop();
+        this.bot.pvp.stop();
+    }
+
+    clearBotLogs() {
+        this.bot.output = '';
+        this.bot.interrupt_code = false;
+    }
+
+    shutUp() {
+        this.shut_up = true;
+        if (this.self_prompter.on) {
+            this.self_prompter.stop(false);
+        }
+        convoManager.endAllConversations();
+    }
+
+    async handleMessage(source, message, max_responses=null) {
+        if (!source || !message) {
+            console.warn('Received empty message from', source);
+            return false;
+        }
+
+        let used_command = false;
+        if (max_responses === null) {
+            max_responses = settings.max_commands === -1 ? Infinity : settings.max_commands;
+        }
+        if (max_responses === -1) {
+            max_responses = Infinity;
+        }
+
+        const self_prompt = source === 'system' || source === this.name;
+        const from_other_bot = convoManager.isOtherAgent(source);
+
+        if (!self_prompt && !from_other_bot) { // from user, check for forced commands
+            const user_command_name = containsCommand(message);
+            if (user_command_name) {
+                if (!commandExists(user_command_name)) {
+                    this.routeResponse(source, `Command '${user_command_name}' does not exist.`);
+                    return false;
+                }
+                this.routeResponse(source, `*${source} used ${user_command_name.substring(1)}*`);
+                if (user_command_name === '!newAction') {
+                    // all user-initiated commands are ignored by the bot except for this one
+                    // add the preceding message to the history to give context for newAction
+                    this.history.add(source, message);
+                }
+                let execute_res = await executeCommand(this, message);
+                if (execute_res) 
+                    this.routeResponse(source, execute_res);
+                return true;
+            }
+        }
+
+        if (from_other_bot)
+            this.last_sender = source;
+
+        // Now translate the message
+        message = await handleEnglishTranslation(message);
+        console.log('received message from', source, ':', message);
+
+        const checkInterrupt = () => this.self_prompter.shouldInterrupt(self_prompt) || this.shut_up || convoManager.responseScheduledFor(source);
+        
+        let behavior_log = this.bot.modes.flushBehaviorLog();
+        if (behavior_log.trim().length > 0) {
+            const MAX_LOG = 500;
+            if (behavior_log.length > MAX_LOG) {
+                behavior_log = '...' + behavior_log.substring(behavior_log.length - MAX_LOG);
+            }
+            behavior_log = 'Recent behaviors log: \n' + behavior_log.substring(behavior_log.indexOf('\n'));
+            await this.history.add('system', behavior_log);
+        }
+
+        // Handle other user messages
+        await this.history.add(source, message);
+        this.history.save();
+
+        if (!self_prompt && this.self_prompter.on) // message is from user during self-prompting
+            max_responses = 1; // force only respond to this message, then let self-prompting take over
+        for (let i=0; i<max_responses; i++) {
+            if (checkInterrupt()) break;
+            let history = this.history.getHistory();
+            let res = await this.prompter.promptConvo(history);
+
+            console.log(`${this.name} full response to ${source}: ""${res}""`);
+            
+            if (res.trim().length === 0) { 
+                console.warn('no response')
+                break; // empty response ends loop
+            }
+
+            let command_name = containsCommand(res);
+
+            if (command_name) { // contains query or command
+                res = truncCommandMessage(res); // everything after the command is ignored
+                this.history.add(this.name, res);
+                
+                if (!commandExists(command_name)) {
+                    this.history.add('system', `Command ${command_name} does not exist.`);
+                    console.warn('Agent hallucinated command:', command_name)
+                    continue;
+                }
+
+                if (checkInterrupt()) break;
+                this.self_prompter.handleUserPromptedCmd(self_prompt, isAction(command_name));
+
+                if (settings.verbose_commands) {
+                    this.routeResponse(source, res);
+                }
+                else { // only output command name
+                    let pre_message = res.substring(0, res.indexOf(command_name)).trim();
+                    let chat_message = `*used ${command_name.substring(1)}*`;
+                    if (pre_message.length > 0)
+                        chat_message = `${pre_message}  ${chat_message}`;
+                    this.routeResponse(source, chat_message);
+                }
+
+                let execute_res = await executeCommand(this, res);
+
+                console.log('Agent executed:', command_name, 'and got:', execute_res);
+                used_command = true;
+
+                if (execute_res)
+                    this.history.add('system', execute_res);
+                else
+                    break;
+            }
+            else { // conversation response
+                this.history.add(this.name, res);
+                this.routeResponse(source, res);
+                break;
+            }
+            
+            this.history.save();
+        }
+
+        return used_command;
+    }
+
+    async routeResponse(to_player, message) {
+        if (this.shut_up) return;
+        let self_prompt = to_player === 'system' || to_player === this.name;
+        if (self_prompt && this.last_sender) {
+            // this is for when the agent is prompted by system while still in conversation
+            // so it can respond to events like death but be routed back to the last sender
+            to_player = this.last_sender;
+        }
+
+        if (convoManager.isOtherAgent(to_player) && convoManager.inConversation(to_player)) {
+            // if we're in an ongoing conversation with the other bot, send the response to it
+            convoManager.sendToBot(to_player, message);
+        }
+        else {
+            // otherwise, use open chat
+            this.openChat(message);
+            // note that to_player could be another bot, but if we get here the conversation has ended
+        }
+    }
+
+    async openChat(message) {
+        let to_translate = message;
+        let remaining = '';
+        let command_name = containsCommand(message);
+        let translate_up_to = command_name ? message.indexOf(command_name) : -1;
+        if (translate_up_to != -1) { // don't translate the command
+            to_translate = to_translate.substring(0, translate_up_to);
+            remaining = message.substring(translate_up_to);
+        }
+        message = (await handleTranslation(to_translate)).trim() + " " + remaining;
+        // newlines are interpreted as separate chats, which triggers spam filters. replace them with spaces
+        message = message.replaceAll('\n', ' ');
+
+        if (settings.only_chat_with.length > 0) {
+            for (let username of settings.only_chat_with) {
+                this.bot.whisper(username, message);
+            }
+        }
+        else {
+            this.bot.chat(message);
+        }
+    }
+
+    startEvents() {
+        // Custom events
+        this.bot.on('time', () => {
+            if (this.bot.time.timeOfDay == 0)
+            this.bot.emit('sunrise');
+            else if (this.bot.time.timeOfDay == 6000)
+            this.bot.emit('noon');
+            else if (this.bot.time.timeOfDay == 12000)
+            this.bot.emit('sunset');
+            else if (this.bot.time.timeOfDay == 18000)
+            this.bot.emit('midnight');
+        });
+
+        let prev_health = this.bot.health;
+        this.bot.lastDamageTime = 0;
+        this.bot.lastDamageTaken = 0;
+        this.bot.on('health', () => {
+            if (this.bot.health < prev_health) {
+                this.bot.lastDamageTime = Date.now();
+                this.bot.lastDamageTaken = prev_health - this.bot.health;
+            }
+            prev_health = this.bot.health;
+        });
+        // Logging callbacks
+        this.bot.on('error' , (err) => {
+            console.error('Error event!', err);
+        });
+        this.bot.on('end', (reason) => {
+            console.warn('Bot disconnected! Killing agent process.', reason)
+            this.cleanKill('Bot disconnected! Killing agent process.');
+        });
+        this.bot.on('death', () => {
+            this.actions.cancelResume();
+            this.actions.stop();
+        });
+        this.bot.on('kicked', (reason) => {
+            console.warn('Bot kicked!', reason);
+            this.cleanKill('Bot kicked! Killing agent process.');
+        });
+        this.bot.on('messagestr', async (message, _, jsonMsg) => {
+            if (jsonMsg.translate && jsonMsg.translate.startsWith('death') && message.startsWith(this.name)) {
+                console.log('Agent died: ', message);
+                let death_pos = this.bot.entity.position;
+                this.memory_bank.rememberPlace('last_death_position', death_pos.x, death_pos.y, death_pos.z);
+                let death_pos_text = null;
+                if (death_pos) {
+                    death_pos_text = `x: ${death_pos.x.toFixed(2)}, y: ${death_pos.y.toFixed(2)}, z: ${death_pos.x.toFixed(2)}`;
+                }
+                let dimention = this.bot.game.dimension;
+                this.handleMessage('system', `You died at position ${death_pos_text || "unknown"} in the ${dimention} dimension with the final message: '${message}'. Your place of death is saved as 'last_death_position' if you want to return. Previous actions were stopped and you have respawned.`);
+            }
+        });
+        this.bot.on('idle', () => {
+            this.bot.clearControlStates();
+            this.bot.pathfinder.stop(); // clear any lingering pathfinder
+            this.bot.modes.unPauseAll();
+            this.actions.resumeAction();
+        });
+
+        // Init NPC controller
+        this.npc.init();
+
+        // This update loop ensures that each update() is called one at a time, even if it takes longer than the interval
+        const INTERVAL = 300;
+        let last = Date.now();
+        setTimeout(async () => {
+            while (true) {
+                let start = Date.now();
+                await this.update(start - last);
+                let remaining = INTERVAL - (Date.now() - start);
+                if (remaining > 0) {
+                    await new Promise((resolve) => setTimeout(resolve, remaining));
+                }
+                last = start;
+            }
+        }, INTERVAL);
+
+        this.bot.emit('idle');
+    }
+
+    async update(delta) {
+        await this.bot.modes.update();
+        this.self_prompter.update(delta);
+        if (this.task.data) {
+            let res = this.task.isDone();
+            if (res) {
+                console.log('Task finished:', res.message);
+                this.killAll();
+            }
+        }
+    }
+
+    isIdle() {
+        return !this.actions.executing && !this.coder.generating;
+    }
+    
+    cleanKill(msg='Killing agent process...', code=1) {
+        this.history.add('system', msg);
+        this.bot.chat(code > 1 ? 'Restarting.': 'Exiting.');
+        this.history.save();
+        process.exit(code);
+    }
+
+    killAll() {
+        serverProxy.shutdown();
+    }
+}

src/agent/agent_proxy.js

@@ -0,0 +1,73 @@
+import { io } from 'socket.io-client';
+import convoManager from './conversation.js';
+import settings from '../../settings.js';
+
+class AgentServerProxy {
+    constructor() {
+        if (AgentServerProxy.instance) {
+            return AgentServerProxy.instance;
+        }
+        
+        this.socket = null;
+        this.connected = false;
+        AgentServerProxy.instance = this;
+    }
+
+    connect(agent) {
+        if (this.connected) return;
+        
+        this.agent = agent;
+
+        this.socket = io(`http://${settings.mindserver_host}:${settings.mindserver_port}`);
+        this.connected = true;
+
+        this.socket.on('connect', () => {
+            console.log('Connected to MindServer');
+        });
+
+        this.socket.on('disconnect', () => {
+            console.log('Disconnected from MindServer');
+            this.connected = false;
+        });
+
+        this.socket.on('chat-message', (agentName, json) => {
+            convoManager.receiveFromBot(agentName, json);
+        });
+
+        this.socket.on('agents-update', (agents) => {
+            convoManager.updateAgents(agents);
+        });
+
+        this.socket.on('restart-agent', (agentName) => {
+            console.log(`Restarting agent: ${agentName}`);
+            this.agent.cleanKill();
+        });
+		
+		this.socket.on('send-message', (agentName, message) => {
+			try {
+				this.agent.respondFunc("NO USERNAME", message);
+			} catch (error) {
+				console.error('Error: ', JSON.stringify(error, Object.getOwnPropertyNames(error)));
+			}
+		});
+    }
+
+    login() {
+        this.socket.emit('login-agent', this.agent.name);
+    }
+
+    shutdown() {
+        this.socket.emit('shutdown');
+    }
+
+    getSocket() {
+        return this.socket;
+    }
+}
+
+// Create and export a singleton instance
+export const serverProxy = new AgentServerProxy();
+
+export function sendBotChatToServer(agentName, json) {
+    serverProxy.getSocket().emit('chat-message', agentName, json);
+}

src/agent/coder.js

@@ -0,0 +1,228 @@
+import { writeFile, readFile, mkdirSync } from 'fs';
+import settings from '../../settings.js';
+import { makeCompartment } from './library/lockdown.js';
+import * as skills from './library/skills.js';
+import * as world from './library/world.js';
+import { Vec3 } from 'vec3';
+import {ESLint} from "eslint";
+
+export class Coder {
+    constructor(agent) {
+        this.agent = agent;
+        this.file_counter = 0;
+        this.fp = '/bots/'+agent.name+'/action-code/';
+        this.generating = false;
+        this.code_template = '';
+        this.code_lint_template = '';
+
+        readFile('./bots/execTemplate.js', 'utf8', (err, data) => {
+            if (err) throw err;
+            this.code_template = data;
+        });
+        readFile('./bots/lintTemplate.js', 'utf8', (err, data) => {
+            if (err) throw err;
+            this.code_lint_template = data;
+        });
+        mkdirSync('.' + this.fp, { recursive: true });
+    }
+    
+    async  lintCode(code) {
+        let result = '#### CODE ERROR INFO ###\n';
+        // Extract everything in the code between the beginning of 'skills./world.' and the '('
+        const skillRegex = /(?:skills|world)\.(.*?)\(/g;
+        const skills = [];
+        let match;
+        while ((match = skillRegex.exec(code)) !== null) {
+            skills.push(match[1]);
+        }
+        const allDocs = await this.agent.prompter.skill_libary.getRelevantSkillDocs();
+        //lint if the function exists
+        const missingSkills = skills.filter(skill => !allDocs.includes(skill));
+        if (missingSkills.length > 0) {
+            result += 'These functions do not exist. Please modify the correct function name and try again.\n';
+            result += '### FUNCTIONS NOT FOUND ###\n';
+            result += missingSkills.join('\n');
+            console.log(result)
+            return result;
+        }
+
+        const eslint = new ESLint();
+        const results = await eslint.lintText(code);
+        const codeLines = code.split('\n');
+        const exceptions = results.map(r => r.messages).flat();
+
+        if (exceptions.length > 0) {
+            exceptions.forEach((exc, index) => {
+                if (exc.line && exc.column ) {
+                    const errorLine = codeLines[exc.line - 1]?.trim() || 'Unable to retrieve error line content';
+                    result += `#ERROR ${index + 1}\n`;
+                    result += `Message: ${exc.message}\n`;
+                    result += `Location: Line ${exc.line}, Column ${exc.column}\n`;
+                    result += `Related Code Line: ${errorLine}\n`;
+                }
+            });
+            result += 'The code contains exceptions and cannot continue execution.';
+        } else {
+            return null;//no error
+        }
+
+        return result ;
+    }
+    // write custom code to file and import it
+    // write custom code to file and prepare for evaluation
+    async stageCode(code) {
+        code = this.sanitizeCode(code);
+        let src = '';
+        code = code.replaceAll('console.log(', 'log(bot,');
+        code = code.replaceAll('log("', 'log(bot,"');
+
+        console.log(`Generated code: """${code}"""`);
+
+        // this may cause problems in callback functions
+        code = code.replaceAll(';\n', '; if(bot.interrupt_code) {log(bot, "Code interrupted.");return;}\n');
+        for (let line of code.split('\n')) {
+            src += `    ${line}\n`;
+        }
+        let src_lint_copy = this.code_lint_template.replace('/* CODE HERE */', src);
+        src = this.code_template.replace('/* CODE HERE */', src);
+
+        let filename = this.file_counter + '.js';
+        // if (this.file_counter > 0) {
+        //     let prev_filename = this.fp + (this.file_counter-1) + '.js';
+        //     unlink(prev_filename, (err) => {
+        //         console.log("deleted file " + prev_filename);
+        //         if (err) console.error(err);
+        //     });
+        // } commented for now, useful to keep files for debugging
+        this.file_counter++;
+        
+        let write_result = await this.writeFilePromise('.' + this.fp + filename, src);
+        // This is where we determine the environment the agent's code should be exposed to.
+        // It will only have access to these things, (in addition to basic javascript objects like Array, Object, etc.)
+        // Note that the code may be able to modify the exposed objects.
+        const compartment = makeCompartment({
+            skills,
+            log: skills.log,
+            world,
+            Vec3,
+        });
+        const mainFn = compartment.evaluate(src);
+        
+        if (write_result) {
+            console.error('Error writing code execution file: ' + result);
+            return null;
+        }
+        return { func:{main: mainFn}, src_lint_copy: src_lint_copy };
+    }
+
+    sanitizeCode(code) {
+        code = code.trim();
+        const remove_strs = ['Javascript', 'javascript', 'js']
+        for (let r of remove_strs) {
+            if (code.startsWith(r)) {
+                code = code.slice(r.length);
+                return code;
+            }
+        }
+        return code;
+    }
+
+    writeFilePromise(filename, src) {
+        // makes it so we can await this function
+        return new Promise((resolve, reject) => {
+            writeFile(filename, src, (err) => {
+                if (err) {
+                    reject(err);
+                } else {
+                    resolve();
+                }
+            });
+        });
+    }
+
+    async generateCode(agent_history) {
+        // wrapper to prevent overlapping code generation loops
+        await this.agent.actions.stop();
+        this.generating = true;
+        let res = await this.generateCodeLoop(agent_history);
+        this.generating = false;
+        if (!res.interrupted) this.agent.bot.emit('idle');
+        return res.message;
+    }
+
+    async generateCodeLoop(agent_history) {
+        this.agent.bot.modes.pause('unstuck');
+
+        let messages = agent_history.getHistory();
+        messages.push({role: 'system', content: 'Code generation started. Write code in codeblock in your response:'});
+
+        let code = null;
+        let code_return = null;
+        let failures = 0;
+        const interrupt_return = {success: true, message: null, interrupted: true, timedout: false};
+        for (let i=0; i<5; i++) {
+            if (this.agent.bot.interrupt_code)
+                return interrupt_return;
+            console.log(messages)
+            let res = await this.agent.prompter.promptCoding(JSON.parse(JSON.stringify(messages)));
+            if (this.agent.bot.interrupt_code)
+                return interrupt_return;
+            let contains_code = res.indexOf('```') !== -1;
+            if (!contains_code) {
+                if (res.indexOf('!newAction') !== -1) {
+                    messages.push({
+                        role: 'assistant', 
+                        content: res.substring(0, res.indexOf('!newAction'))
+                    });
+                    continue; // using newaction will continue the loop
+                }
+                
+                if (failures >= 3) {
+                    return { success: false, message: 'Action failed, agent would not write code.', interrupted: false, timedout: false };
+                }
+                messages.push({
+                    role: 'system', 
+                    content: 'Error: no code provided. Write code in codeblock in your response. ``` // example ```'}
+                );
+                failures++;
+                continue;
+            }
+            code = res.substring(res.indexOf('```')+3, res.lastIndexOf('```'));
+            const result = await this.stageCode(code);
+            const executionModuleExports = result.func;
+            let src_lint_copy = result.src_lint_copy;
+            const analysisResult = await this.lintCode(src_lint_copy);
+            if (analysisResult) {
+                const message = 'Error: Code syntax error. Please try again:'+'\n'+analysisResult+'\n';
+                messages.push({ role: 'system', content: message });
+                continue;
+            }
+            if (!executionModuleExports) {
+                agent_history.add('system', 'Failed to stage code, something is wrong.');
+                return {success: false, message: null, interrupted: false, timedout: false};
+            }
+            
+            code_return = await this.agent.actions.runAction('newAction', async () => {
+                return await executionModuleExports.main(this.agent.bot);
+            }, { timeout: settings.code_timeout_mins });
+            if (code_return.interrupted && !code_return.timedout)
+                return { success: false, message: null, interrupted: true, timedout: false };
+            console.log("Code generation result:", code_return.success, code_return.message.toString());
+
+            if (code_return.success) {
+                const summary = "Summary of newAction\nAgent wrote this code: \n```" + this.sanitizeCode(code) + "```\nCode Output:\n" + code_return.message.toString();
+                return { success: true, message: summary, interrupted: false, timedout: false };
+            }
+
+            messages.push({
+                role: 'assistant',
+                content: res
+            });
+            messages.push({
+                role: 'system',
+                content: code_return.message + '\nCode failed. Please try again:'
+            });
+        }
+        return { success: false, message: null, interrupted: false, timedout: true };
+    }
+}

src/agent/commands/actions.js

@@ -0,0 +1,423 @@
+import * as skills from '../library/skills.js';
+import settings from '../../../settings.js';
+import convoManager from '../conversation.js';
+
+function runAsAction (actionFn, resume = false, timeout = -1) {
+    let actionLabel = null;  // Will be set on first use
+    
+    const wrappedAction = async function (agent, ...args) {
+        // Set actionLabel only once, when the action is first created
+        if (!actionLabel) {
+            const actionObj = actionsList.find(a => a.perform === wrappedAction);
+            actionLabel = actionObj.name.substring(1); // Remove the ! prefix
+        }
+
+        const actionFnWithAgent = async () => {
+            await actionFn(agent, ...args);
+        };
+        const code_return = await agent.actions.runAction(`action:${actionLabel}`, actionFnWithAgent, { timeout, resume });
+        if (code_return.interrupted && !code_return.timedout)
+            return;
+        return code_return.message;
+    }
+
+    return wrappedAction;
+}
+
+export const actionsList = [
+    {
+        name: '!newAction',
+        description: 'Perform new and unknown custom behaviors that are not available as a command.', 
+        params: {
+            'prompt': { type: 'string', description: 'A natural language prompt to guide code generation. Make a detailed step-by-step plan.' }
+        },
+        perform: async function (agent, prompt) {
+            // just ignore prompt - it is now in context in chat history
+            if (!settings.allow_insecure_coding)
+                return 'newAction not allowed! Code writing is disabled in settings. Notify the user.';
+            return await agent.coder.generateCode(agent.history);
+        }
+    },
+    {
+        name: '!stop',
+        description: 'Force stop all actions and commands that are currently executing.',
+        perform: async function (agent) {
+            await agent.actions.stop();
+            agent.clearBotLogs();
+            agent.actions.cancelResume();
+            agent.bot.emit('idle');
+            let msg = 'Agent stopped.';
+            if (agent.self_prompter.on)
+                msg += ' Self-prompting still active.';
+            return msg;
+        }
+    },
+    {
+        name: '!stfu',
+        description: 'Stop all chatting and self prompting, but continue current action.',
+        perform: async function (agent) {
+            agent.openChat('Shutting up.');
+            agent.shutUp();
+            return;
+        }
+    },
+    {
+        name: '!restart',
+        description: 'Restart the agent process.',
+        perform: async function (agent) {
+            agent.cleanKill();
+        }
+    },
+    {
+        name: '!clearChat',
+        description: 'Clear the chat history.',
+        perform: async function (agent) {
+            agent.history.clear();
+            return agent.name + "'s chat history was cleared, starting new conversation from scratch.";
+        }
+    },
+    {
+        name: '!goToPlayer',
+        description: 'Go to the given player.',
+        params: {
+            'player_name': {type: 'string', description: 'The name of the player to go to.'},
+            'closeness': {type: 'float', description: 'How close to get to the player.', domain: [0, Infinity]}
+        },
+        perform: runAsAction(async (agent, player_name, closeness) => {
+            return await skills.goToPlayer(agent.bot, player_name, closeness);
+        })
+    },
+    {
+        name: '!followPlayer',
+        description: 'Endlessly follow the given player.',
+        params: {
+            'player_name': {type: 'string', description: 'name of the player to follow.'},
+            'follow_dist': {type: 'float', description: 'The distance to follow from.', domain: [0, Infinity]}
+        },
+        perform: runAsAction(async (agent, player_name, follow_dist) => {
+            await skills.followPlayer(agent.bot, player_name, follow_dist);
+        }, true)
+    },
+    {
+        name: '!goToCoordinates',
+        description: 'Go to the given x, y, z location.',
+        params: {
+            'x': {type: 'float', description: 'The x coordinate.', domain: [-Infinity, Infinity]},
+            'y': {type: 'float', description: 'The y coordinate.', domain: [-64, 320]},
+            'z': {type: 'float', description: 'The z coordinate.', domain: [-Infinity, Infinity]},
+            'closeness': {type: 'float', description: 'How close to get to the location.', domain: [0, Infinity]}
+        },
+        perform: runAsAction(async (agent, x, y, z, closeness) => {
+            await skills.goToPosition(agent.bot, x, y, z, closeness);
+        })
+    },
+    {
+        name: '!searchForBlock',
+        description: 'Find and go to the nearest block of a given type in a given range.',
+        params: {
+            'type': { type: 'BlockName', description: 'The block type to go to.' },
+            'search_range': { type: 'float', description: 'The range to search for the block.', domain: [32, 512] }
+        },
+        perform: runAsAction(async (agent, block_type, range) => {
+            await skills.goToNearestBlock(agent.bot, block_type, 4, range);
+        })
+    },
+    {
+        name: '!searchForEntity',
+        description: 'Find and go to the nearest entity of a given type in a given range.',
+        params: {
+            'type': { type: 'string', description: 'The type of entity to go to.' },
+            'search_range': { type: 'float', description: 'The range to search for the entity.', domain: [32, 512] }
+        },
+        perform: runAsAction(async (agent, entity_type, range) => {
+            await skills.goToNearestEntity(agent.bot, entity_type, 4, range);
+        })
+    },
+    {
+        name: '!moveAway',
+        description: 'Move away from the current location in any direction by a given distance.',
+        params: {'distance': { type: 'float', description: 'The distance to move away.', domain: [0, Infinity] }},
+        perform: runAsAction(async (agent, distance) => {
+            await skills.moveAway(agent.bot, distance);
+        })
+    },
+    {
+        name: '!rememberHere',
+        description: 'Save the current location with a given name.',
+        params: {'name': { type: 'string', description: 'The name to remember the location as.' }},
+        perform: async function (agent, name) {
+            const pos = agent.bot.entity.position;
+            agent.memory_bank.rememberPlace(name, pos.x, pos.y, pos.z);
+            return `Location saved as "${name}".`;
+        }
+    },
+    {
+        name: '!goToRememberedPlace',
+        description: 'Go to a saved location.',
+        params: {'name': { type: 'string', description: 'The name of the location to go to.' }},
+        perform: runAsAction(async (agent, name) => {
+            const pos = agent.memory_bank.recallPlace(name);
+            if (!pos) {
+            skills.log(agent.bot, `No location named "${name}" saved.`);
+            return;
+            }
+            await skills.goToPosition(agent.bot, pos[0], pos[1], pos[2], 1);
+        })
+    },
+    {
+        name: '!givePlayer',
+        description: 'Give the specified item to the given player.',
+        params: { 
+            'player_name': { type: 'string', description: 'The name of the player to give the item to.' }, 
+            'item_name': { type: 'ItemName', description: 'The name of the item to give.' },
+            'num': { type: 'int', description: 'The number of items to give.', domain: [1, Number.MAX_SAFE_INTEGER] }
+        },
+        perform: runAsAction(async (agent, player_name, item_name, num) => {
+            await skills.giveToPlayer(agent.bot, item_name, player_name, num);
+        })
+    },
+    {
+        name: '!consume',
+        description: 'Eat/drink the given item.',
+        params: {'item_name': { type: 'ItemName', description: 'The name of the item to consume.' }},
+        perform: runAsAction(async (agent, item_name) => {
+            await skills.consume(agent.bot, item_name);
+        })
+    },
+    {
+        name: '!equip',
+        description: 'Equip the given item.',
+        params: {'item_name': { type: 'ItemName', description: 'The name of the item to equip.' }},
+        perform: runAsAction(async (agent, item_name) => {
+            await skills.equip(agent.bot, item_name);
+        })
+    },
+    {
+        name: '!putInChest',
+        description: 'Put the given item in the nearest chest.',
+        params: {
+            'item_name': { type: 'ItemName', description: 'The name of the item to put in the chest.' },
+            'num': { type: 'int', description: 'The number of items to put in the chest.', domain: [1, Number.MAX_SAFE_INTEGER] }
+        },
+        perform: runAsAction(async (agent, item_name, num) => {
+            await skills.putInChest(agent.bot, item_name, num);
+        })
+    },
+    {
+        name: '!takeFromChest',
+        description: 'Take the given items from the nearest chest.',
+        params: {
+            'item_name': { type: 'ItemName', description: 'The name of the item to take.' },
+            'num': { type: 'int', description: 'The number of items to take.', domain: [1, Number.MAX_SAFE_INTEGER] }
+        },
+        perform: runAsAction(async (agent, item_name, num) => {
+            await skills.takeFromChest(agent.bot, item_name, num);
+        })
+    },
+    {
+        name: '!viewChest',
+        description: 'View the items/counts of the nearest chest.',
+        params: { },
+        perform: runAsAction(async (agent) => {
+            await skills.viewChest(agent.bot);
+        })
+    },
+    {
+        name: '!discard',
+        description: 'Discard the given item from the inventory.',
+        params: {
+            'item_name': { type: 'ItemName', description: 'The name of the item to discard.' },
+            'num': { type: 'int', description: 'The number of items to discard.', domain: [1, Number.MAX_SAFE_INTEGER] }
+        },
+        perform: runAsAction(async (agent, item_name, num) => {
+            const start_loc = agent.bot.entity.position;
+            await skills.moveAway(agent.bot, 5);
+            await skills.discard(agent.bot, item_name, num);
+            await skills.goToPosition(agent.bot, start_loc.x, start_loc.y, start_loc.z, 0);
+        })
+    },
+    {
+        name: '!collectBlocks',
+        description: 'Collect the nearest blocks of a given type.',
+        params: {
+            'type': { type: 'BlockName', description: 'The block type to collect.' },
+            'num': { type: 'int', description: 'The number of blocks to collect.', domain: [1, Number.MAX_SAFE_INTEGER] }
+        },
+        perform: runAsAction(async (agent, type, num) => {
+            await skills.collectBlock(agent.bot, type, num);
+        }, false, 10) // 10 minute timeout
+    },
+    {
+        name: '!craftRecipe',
+        description: 'Craft the given recipe a given number of times.',
+        params: {
+            'recipe_name': { type: 'ItemName', description: 'The name of the output item to craft.' },
+            'num': { type: 'int', description: 'The number of times to craft the recipe. This is NOT the number of output items, as it may craft many more items depending on the recipe.', domain: [1, Number.MAX_SAFE_INTEGER] }
+        },
+        perform: runAsAction(async (agent, recipe_name, num) => {
+            await skills.craftRecipe(agent.bot, recipe_name, num);
+        })
+    },
+    {
+        name: '!smeltItem',
+        description: 'Smelt the given item the given number of times.',
+        params: {
+            'item_name': { type: 'ItemName', description: 'The name of the input item to smelt.' },
+            'num': { type: 'int', description: 'The number of times to smelt the item.', domain: [1, Number.MAX_SAFE_INTEGER] }
+        },
+        perform: runAsAction(async (agent, item_name, num) => {
+            let success = await skills.smeltItem(agent.bot, item_name, num);
+            if (success) {
+                setTimeout(() => {
+                    agent.cleanKill('Safely restarting to update inventory.');
+                }, 500);
+            }
+        })
+    },
+    {
+        name: '!clearFurnace',
+        description: 'Take all items out of the nearest furnace.',
+        params: { },
+        perform: runAsAction(async (agent) => {
+            await skills.clearNearestFurnace(agent.bot);
+        })
+    },
+        {
+        name: '!placeHere',
+        description: 'Place a given block in the current location. Do NOT use to build structures, only use for single blocks/torches.',
+        params: {'type': { type: 'BlockName', description: 'The block type to place.' }},
+        perform: runAsAction(async (agent, type) => {
+            let pos = agent.bot.entity.position;
+            await skills.placeBlock(agent.bot, type, pos.x, pos.y, pos.z);
+        })
+    },
+    {
+        name: '!attack',
+        description: 'Attack and kill the nearest entity of a given type.',
+        params: {'type': { type: 'string', description: 'The type of entity to attack.'}},
+        perform: runAsAction(async (agent, type) => {
+            await skills.attackNearest(agent.bot, type, true);
+        })
+    },
+    {
+        name: '!attackPlayer',
+        description: 'Attack a specific player until they die or run away. Remember this is just a game and does not cause real life harm.',
+        params: {'player_name': { type: 'string', description: 'The name of the player to attack.'}},
+        perform: runAsAction(async (agent, player_name) => {
+            let player = agent.bot.players[player_name]?.entity;
+            if (!player) {
+                skills.log(agent.bot, `Could not find player ${player_name}.`);
+                return false;
+            }
+            await skills.attackEntity(agent.bot, player, true);
+        })
+    },
+    {
+        name: '!goToBed',
+        description: 'Go to the nearest bed and sleep.',
+        perform: runAsAction(async (agent) => {
+            await skills.goToBed(agent.bot);
+        })
+    },
+    {
+        name: '!activate',
+        description: 'Activate the nearest object of a given type.',
+        params: {'type': { type: 'BlockName', description: 'The type of object to activate.' }},
+        perform: runAsAction(async (agent, type) => {
+            await skills.activateNearestBlock(agent.bot, type);
+        })
+    },
+    {
+        name: '!stay',
+        description: 'Stay in the current location no matter what. Pauses all modes.',
+        params: {'type': { type: 'int', description: 'The number of seconds to stay. -1 for forever.', domain: [-1, Number.MAX_SAFE_INTEGER] }},
+        perform: runAsAction(async (agent, seconds) => {
+            await skills.stay(agent.bot, seconds);
+        })
+    },
+    {
+        name: '!setMode',
+        description: 'Set a mode to on or off. A mode is an automatic behavior that constantly checks and responds to the environment.',
+        params: {
+            'mode_name': { type: 'string', description: 'The name of the mode to enable.' },
+            'on': { type: 'boolean', description: 'Whether to enable or disable the mode.' }
+        },
+        perform: async function (agent, mode_name, on) {
+            const modes = agent.bot.modes;
+            if (!modes.exists(mode_name))
+            return `Mode ${mode_name} does not exist.` + modes.getDocs();
+            if (modes.isOn(mode_name) === on)
+            return `Mode ${mode_name} is already ${on ? 'on' : 'off'}.`;
+            modes.setOn(mode_name, on);
+            return `Mode ${mode_name} is now ${on ? 'on' : 'off'}.`;
+        }
+    },
+    {
+        name: '!goal',
+        description: 'Set a goal prompt to endlessly work towards with continuous self-prompting.',
+        params: {
+            'selfPrompt': { type: 'string', description: 'The goal prompt.' },
+        },
+        perform: async function (agent, prompt) {
+            if (convoManager.inConversation()) {
+                agent.self_prompter.setPrompt(prompt);
+                convoManager.scheduleSelfPrompter();
+            }
+            else {
+                agent.self_prompter.start(prompt);
+            }
+        }
+    },
+    {
+        name: '!endGoal',
+        description: 'Call when you have accomplished your goal. It will stop self-prompting and the current action. ',
+        perform: async function (agent) {
+            agent.self_prompter.stop();
+            convoManager.cancelSelfPrompter();
+            return 'Self-prompting stopped.';
+        }
+    },
+    {
+        name: '!startConversation',
+        description: 'Start a conversation with a player. Use for bots only.',
+        params: {
+            'player_name': { type: 'string', description: 'The name of the player to send the message to.' },
+            'message': { type: 'string', description: 'The message to send.' },
+        },
+        perform: async function (agent, player_name, message) {
+            if (!convoManager.isOtherAgent(player_name))
+                return player_name + ' is not a bot, cannot start conversation.';
+            if (convoManager.inConversation() && !convoManager.inConversation(player_name)) 
+                convoManager.forceEndCurrentConversation();
+            else if (convoManager.inConversation(player_name))
+                agent.history.add('system', 'You are already in conversation with ' + player_name + '. Don\'t use this command to talk to them.');
+            convoManager.startConversation(player_name, message);
+        }
+    },
+    {
+        name: '!endConversation',
+        description: 'End the conversation with the given player.',
+        params: {
+            'player_name': { type: 'string', description: 'The name of the player to end the conversation with.' }
+        },
+        perform: async function (agent, player_name) {
+            if (!convoManager.inConversation(player_name))
+                return `Not in conversation with ${player_name}.`;
+            convoManager.endConversation(player_name);
+            return `Converstaion with ${player_name} ended.`;
+        }
+    },
+    // { // commented for now, causes confusion with goal command
+    //     name: '!npcGoal',
+    //     description: 'Set a simple goal for an item or building to automatically work towards. Do not use for complex goals.',
+    //     params: {
+    //         'name': { type: 'string', description: 'The name of the goal to set. Can be item or building name. If empty will automatically choose a goal.' },
+    //         'quantity': { type: 'int', description: 'The quantity of the goal to set. Default is 1.', domain: [1, Number.MAX_SAFE_INTEGER] }
+    //     },
+    //     perform: async function (agent, name=null, quantity=1) {
+    //         await agent.npc.setGoal(name, quantity);
+    //         agent.bot.emit('idle');  // to trigger the goal
+    //         return 'Set npc goal: ' + agent.npc.data.curr_goal.name;
+    //     }
+    // },
+];

src/agent/commands/index.js

@@ -0,0 +1,252 @@
+import { getBlockId, getItemId } from "../../utils/mcdata.js";
+import { actionsList } from './actions.js';
+import { queryList } from './queries.js';
+
+let suppressNoDomainWarning = false;
+
+const commandList = queryList.concat(actionsList);
+const commandMap = {};
+for (let command of commandList) {
+    commandMap[command.name] = command;
+}
+
+export function getCommand(name) {
+    return commandMap[name];
+}
+
+export function blacklistCommands(commands) {
+    const unblockable = ['!stop', '!stats', '!inventory', '!goal'];
+    for (let command_name of commands) {
+        if (unblockable.includes(command_name)){
+            console.warn(`Command ${command_name} is unblockable`);
+            continue;
+        }
+        delete commandMap[command_name];
+        delete commandList.find(command => command.name === command_name);
+    }
+}
+
+const commandRegex = /!(\w+)(?:\(((?:-?\d+(?:\.\d+)?|true|false|"[^"]*")(?:\s*,\s*(?:-?\d+(?:\.\d+)?|true|false|"[^"]*"))*)\))?/
+const argRegex = /-?\d+(?:\.\d+)?|true|false|"[^"]*"/g;
+
+export function containsCommand(message) {
+    const commandMatch = message.match(commandRegex);
+    if (commandMatch)
+        return "!" + commandMatch[1];
+    return null;
+}
+
+export function commandExists(commandName) {
+    if (!commandName.startsWith("!"))
+        commandName = "!" + commandName;
+    return commandMap[commandName] !== undefined;
+}
+
+/**
+ * Converts a string into a boolean.
+ * @param {string} input
+ * @returns {boolean | null} the boolean or `null` if it could not be parsed.
+ * */
+function parseBoolean(input) {
+    switch(input.toLowerCase()) {
+        case 'false': //These are interpreted as flase;
+        case 'f':
+        case '0':
+        case 'off':
+            return false;
+        case 'true': //These are interpreted as true;
+        case 't':
+        case '1':
+        case 'on':
+            return true;
+        default:
+            return null;
+    }
+}
+
+/**
+ * @param {number} value - the value to check
+ * @param {number} lowerBound
+ * @param {number} upperBound
+ * @param {string} endpointType - The type of the endpoints represented as a two character string. `'[)'` `'()'` 
+ */
+function checkInInterval(number, lowerBound, upperBound, endpointType) {
+    switch (endpointType) {
+        case '[)':
+            return lowerBound <= number && number < upperBound;
+        case '()':
+            return lowerBound < number && number < upperBound;
+        case '(]':
+            return lowerBound < number && number <= upperBound;
+        case '[]':
+            return lowerBound <= number && number <= upperBound;
+        default:
+            throw new Error('Unknown endpoint type:', endpointType)
+    }
+}
+
+
+
+// todo: handle arrays?
+/**
+ * Returns an object containing the command, the command name, and the comand parameters.
+ * If parsing unsuccessful, returns an error message as a string.
+ * @param {string} message - A message from a player or language model containing a command.
+ * @returns {string | Object}
+ */
+export function parseCommandMessage(message) {
+    const commandMatch = message.match(commandRegex);
+    if (!commandMatch) return `Command is incorrectly formatted`;
+
+    const commandName = "!"+commandMatch[1];
+
+    let args;
+    if (commandMatch[2]) args = commandMatch[2].match(argRegex);
+    else args = [];
+
+    const command = getCommand(commandName);
+    if(!command) return `${commandName} is not a command.`
+
+    const params = commandParams(command);
+    const paramNames = commandParamNames(command);
+    
+    if (args.length !== params.length)
+        return `Command ${command.name} was given ${args.length} args, but requires ${params.length} args.`;
+
+    
+    for (let i = 0; i < args.length; i++) {
+        const param = params[i];
+        //Remove any extra characters
+        let arg = args[i].trim();
+        if ((arg.startsWith('"') && arg.endsWith('"')) || (arg.startsWith("'") && arg.endsWith("'"))) {
+            arg = arg.substring(1, arg.length-1);
+        }
+        
+        //Convert to the correct type
+        switch(param.type) {
+            case 'int':
+                arg = Number.parseInt(arg); break;
+            case 'float':
+                arg = Number.parseFloat(arg); break;
+            case 'boolean':
+                arg = parseBoolean(arg); break;
+            case 'BlockName':
+            case 'ItemName':
+                if (arg.endsWith('plank'))
+                    arg += 's'; // catches common mistakes like "oak_plank" instead of "oak_planks"
+            case 'string':
+                break;
+            default:
+                throw new Error(`Command '${commandName}' parameter '${paramNames[i]}' has an unknown type: ${param.type}`);
+        }
+        if(arg === null || Number.isNaN(arg))
+            return `Error: Param '${paramNames[i]}' must be of type ${param.type}.`
+
+        if(typeof arg === 'number') { //Check the domain of numbers
+            const domain = param.domain;
+            if(domain) {
+                /**
+                 * Javascript has a built in object for sets but not intervals.
+                 * Currently the interval (lowerbound,upperbound] is represented as an Array: `[lowerbound, upperbound, '(]']`
+                 */
+                if (!domain[2]) domain[2] = '[)'; //By default, lower bound is included. Upper is not.
+
+                if(!checkInInterval(arg, ...domain)) {
+                    return `Error: Param '${paramNames[i]}' must be an element of ${domain[2][0]}${domain[0]}, ${domain[1]}${domain[2][1]}.`;
+                    //Alternatively arg could be set to the nearest value in the domain.
+                }
+            } else if (!suppressNoDomainWarning) {
+                console.warn(`Command '${commandName}' parameter '${paramNames[i]}' has no domain set. Expect any value [-Infinity, Infinity].`)
+                suppressNoDomainWarning = true; //Don't spam console. Only give the warning once.
+            }
+        } else if(param.type === 'BlockName') { //Check that there is a block with this name
+            if(getBlockId(arg) == null && arg !== 'air') return  `Invalid block type: ${arg}.`
+        } else if(param.type === 'ItemName') { //Check that there is an item with this name
+            if(getItemId(arg) == null) return `Invalid item type: ${arg}.`
+        }
+        args[i] = arg;
+    }
+    
+    return { commandName, args };
+}
+
+export function truncCommandMessage(message) {
+    const commandMatch = message.match(commandRegex);
+    if (commandMatch) {
+        return message.substring(0, commandMatch.index + commandMatch[0].length);
+    }
+    return message;
+}
+
+export function isAction(name) {
+    return actionsList.find(action => action.name === name) !== undefined;
+}
+
+/**
+ * @param {Object} command
+ * @returns {Object[]} The command's parameters.
+ */
+function commandParams(command) {
+    if (!command.params)
+        return [];
+    return Object.values(command.params);
+}
+
+/**
+ * @param {Object} command
+ * @returns {string[]} The names of the command's parameters.
+ */
+function commandParamNames(command) {
+    if (!command.params)
+        return [];
+    return Object.keys(command.params);
+}
+
+function numParams(command) {
+    return commandParams(command).length;
+}
+
+export async function executeCommand(agent, message) {
+    let parsed = parseCommandMessage(message);
+    if (typeof parsed === 'string')
+        return parsed; //The command was incorrectly formatted or an invalid input was given.
+    else {
+        console.log('parsed command:', parsed);
+        const command = getCommand(parsed.commandName);
+        let numArgs = 0;
+        if (parsed.args) {
+            numArgs = parsed.args.length;
+        }
+        if (numArgs !== numParams(command))
+            return `Command ${command.name} was given ${numArgs} args, but requires ${numParams(command)} args.`;
+        else {
+            const result = await command.perform(agent, ...parsed.args);
+            return result;
+        }
+    }
+}
+
+export function getCommandDocs() {
+    const typeTranslations = {
+        //This was added to keep the prompt the same as before type checks were implemented.
+        //If the language model is giving invalid inputs changing this might help.
+        'float':        'number',
+        'int':          'number',
+        'BlockName':    'string',
+        'ItemName':     'string',
+        'boolean':      'bool'
+    }
+    let docs = `\n*COMMAND DOCS\n You can use the following commands to perform actions and get information about the world. 
+    Use the commands with the syntax: !commandName or !commandName("arg1", 1.2, ...) if the command takes arguments.\n
+    Do not use codeblocks. Use double quotes for strings. Only use one command in each response, trailing commands and comments will be ignored.\n`;
+    for (let command of commandList) {
+        docs += command.name + ': ' + command.description + '\n';
+        if (command.params) {
+            docs += 'Params:\n';
+            for (let param in command.params) {
+                docs += `${param}: (${typeTranslations[command.params[param].type]??command.params[param].type}) ${command.params[param].description}\n`;
+            }
+        }
+    }
+    return docs + '*\n';
+}

src/agent/commands/queries.js

@@ -0,0 +1,224 @@
+import * as world from '../library/world.js';
+import * as mc from '../../utils/mcdata.js';
+import { getCommandDocs } from './index.js';
+import convoManager from '../conversation.js';
+
+const pad = (str) => {
+    return '\n' + str + '\n';
+}
+
+// queries are commands that just return strings and don't affect anything in the world
+export const queryList = [
+    {
+        name: "!stats",
+        description: "Get your bot's location, health, hunger, and time of day.", 
+        perform: function (agent) {
+            let bot = agent.bot;
+            let res = 'STATS';
+            let pos = bot.entity.position;
+            // display position to 2 decimal places
+            res += `\n- Position: x: ${pos.x.toFixed(2)}, y: ${pos.y.toFixed(2)}, z: ${pos.z.toFixed(2)}`;
+            // Gameplay
+            res += `\n- Gamemode: ${bot.game.gameMode}`;
+            res += `\n- Health: ${Math.round(bot.health)} / 20`;
+            res += `\n- Hunger: ${Math.round(bot.food)} / 20`;
+            res += `\n- Biome: ${world.getBiomeName(bot)}`;
+            let weather = "Clear";
+            if (bot.rainState > 0)
+                weather = "Rain";
+            if (bot.thunderState > 0)
+                weather = "Thunderstorm";
+            res += `\n- Weather: ${weather}`;
+            // let block = bot.blockAt(pos);
+            // res += `\n- Artficial light: ${block.skyLight}`;
+            // res += `\n- Sky light: ${block.light}`;
+            // light properties are bugged, they are not accurate
+            res += '\n- ' + world.getSurroundingBlocks(bot).join('\n- ')
+            res += `\n- First Solid Block Above Head: ${world.getFirstBlockAboveHead(bot, null, 32)}`;
+
+
+            if (bot.time.timeOfDay < 6000) {
+                res += '\n- Time: Morning';
+            } else if (bot.time.timeOfDay < 12000) {
+                res += '\n- Time: Afternoon';
+            } else {
+                res += '\n- Time: Night';
+            }
+
+            // get the bot's current action
+            let action = agent.actions.currentActionLabel;
+            if (agent.isIdle())
+                action = 'Idle';
+            res += `\- Current Action: ${action}`;
+
+
+            let players = world.getNearbyPlayerNames(bot);
+            let bots = convoManager.getInGameAgents().filter(b => b !== agent.name);
+            players = players.filter(p => !bots.includes(p));
+
+            res += '\n- Nearby Human Players: ' + (players.length > 0 ? players.join(', ') : 'None.');
+            res += '\n- Nearby Bot Players: ' + (bots.length > 0 ? bots.join(', ') : 'None.');
+
+            res += '\n' + agent.bot.modes.getMiniDocs() + '\n';
+            return pad(res);
+        }
+    },
+    {
+        name: "!inventory",
+        description: "Get your bot's inventory.",
+        perform: function (agent) {
+            let bot = agent.bot;
+            let inventory = world.getInventoryCounts(bot);
+            let res = 'INVENTORY';
+            for (const item in inventory) {
+                if (inventory[item] && inventory[item] > 0)
+                    res += `\n- ${item}: ${inventory[item]}`;
+            }
+            if (res === 'INVENTORY') {
+                res += ': Nothing';
+            }
+            else if (agent.bot.game.gameMode === 'creative') {
+                res += '\n(You have infinite items in creative mode. You do not need to gather resources!!)';
+            }
+
+            let helmet = bot.inventory.slots[5];
+            let chestplate = bot.inventory.slots[6];
+            let leggings = bot.inventory.slots[7];
+            let boots = bot.inventory.slots[8];
+            res += '\nWEARING: ';
+            if (helmet)
+                res += `\nHead: ${helmet.name}`;
+            if (chestplate)
+                res += `\nTorso: ${chestplate.name}`;
+            if (leggings)
+                res += `\nLegs: ${leggings.name}`;
+            if (boots)
+                res += `\nFeet: ${boots.name}`;
+            if (!helmet && !chestplate && !leggings && !boots)
+                res += 'Nothing';
+
+            return pad(res);
+        }
+    },
+    {
+        name: "!nearbyBlocks",
+        description: "Get the blocks near the bot.",
+        perform: function (agent) {
+            let bot = agent.bot;
+            let res = 'NEARBY_BLOCKS';
+            let blocks = world.getNearbyBlockTypes(bot);
+            for (let i = 0; i < blocks.length; i++) {
+                res += `\n- ${blocks[i]}`;
+            }
+            if (blocks.length == 0) {
+                res += ': none';
+            } 
+            else {
+                // Environmental Awareness
+                res += '\n- ' + world.getSurroundingBlocks(bot).join('\n- ')
+                res += `\n- First Solid Block Above Head: ${world.getFirstBlockAboveHead(bot, null, 32)}`;
+            }
+            return pad(res);
+        }
+    },
+    {
+        name: "!craftable",
+        description: "Get the craftable items with the bot's inventory.",
+        perform: function (agent) {
+            let craftable = world.getCraftableItems(agent.bot);
+            let res = 'CRAFTABLE_ITEMS';
+            for (const item of craftable) {
+                res += `\n- ${item}`;
+            }
+            if (res == 'CRAFTABLE_ITEMS') {
+                res += ': none';
+            }
+            return pad(res);
+        }
+    },
+    {
+        name: "!entities",
+        description: "Get the nearby players and entities.",
+        perform: function (agent) {
+            let bot = agent.bot;
+            let res = 'NEARBY_ENTITIES';
+            let players = world.getNearbyPlayerNames(bot);
+            let bots = convoManager.getInGameAgents().filter(b => b !== agent.name);
+            players = players.filter(p => !bots.includes(p));
+
+            for (const player of players) {
+                res += `\n- Human player: ${player}`;
+            }
+            for (const bot of bots) {
+                res += `\n- Bot player: ${bot}`;
+            }
+
+            for (const entity of world.getNearbyEntityTypes(bot)) {
+                if (entity === 'player' || entity === 'item')
+                    continue;
+                res += `\n- entities: ${entity}`;
+            }
+            if (res == 'NEARBY_ENTITIES') {
+                res += ': none';
+            }
+            return pad(res);
+        }
+    },
+    {
+        name: "!modes",
+        description: "Get all available modes and their docs and see which are on/off.",
+        perform: function (agent) {
+            return agent.bot.modes.getDocs();
+        }
+    },
+    {
+        name: '!savedPlaces',
+        description: 'List all saved locations.',
+        perform: async function (agent) {
+            return "Saved place names: " + agent.memory_bank.getKeys();
+        }
+    },
+    {
+        name: '!getCraftingPlan',
+        description: "Provides a comprehensive crafting plan for a specified item. This includes a breakdown of required ingredients, the exact quantities needed, and an analysis of missing ingredients or extra items needed based on the bot's current inventory.",
+        params: {
+            targetItem: { 
+                type: 'string', 
+                description: 'The item that we are trying to craft' 
+            },
+            quantity: { 
+                type: 'int',
+                description: 'The quantity of the item that we are trying to craft',
+                optional: true,
+                domain: [1, Infinity, '[)'], // Quantity must be at least 1,
+                default: 1
+            }
+        },
+        perform: function (agent, targetItem, quantity = 1) {
+            let bot = agent.bot;
+
+            // Fetch the bot's inventory
+            const curr_inventory = world.getInventoryCounts(bot); 
+            const target_item = targetItem;
+            let existingCount = curr_inventory[target_item] || 0;
+            let prefixMessage = '';
+            if (existingCount > 0) {
+                curr_inventory[target_item] -= existingCount;
+                prefixMessage = `You already have ${existingCount} ${target_item} in your inventory. If you need to craft more,\n`;
+            }
+
+            // Generate crafting plan
+            let craftingPlan = mc.getDetailedCraftingPlan(target_item, quantity, curr_inventory);
+            craftingPlan = prefixMessage + craftingPlan;
+            console.log(craftingPlan);
+            return pad(craftingPlan);
+        },
+    },
+    {
+        name: '!help',
+        description: 'Lists all available commands and their descriptions.',
+        perform: async function (agent) {
+            return getCommandDocs();
+        }
+    },
+];

src/agent/conversation.js

@@ -0,0 +1,367 @@
+import settings from '../../settings.js';
+import { readFileSync } from 'fs';
+import { containsCommand } from './commands/index.js';
+import { sendBotChatToServer } from './agent_proxy.js';
+
+let agent;
+let agent_names = settings.profiles.map((p) => JSON.parse(readFileSync(p, 'utf8')).name);
+let agents_in_game = [];
+
+let self_prompter_paused = false;
+
+class Conversation {
+    constructor(name) {
+        this.name = name;
+        this.active = false;
+        this.ignore_until_start = false;
+        this.blocked = false;
+        this.in_queue = [];
+        this.inMessageTimer = null;
+    }
+
+    reset() {
+        this.active = false;
+        this.ignore_until_start = false;
+        this.in_queue = [];
+        this.inMessageTimer = null;
+    }
+
+    end() {
+        this.active = false;
+        this.ignore_until_start = true;
+        this.inMessageTimer = null;
+        const full_message = _compileInMessages(this);
+        if (full_message.message.trim().length > 0)
+            agent.history.add(this.name, full_message.message);
+        // add the full queued messages to history, but don't respond
+
+        if (agent.last_sender === this.name)
+            agent.last_sender = null;
+    }
+
+    queue(message) {
+        this.in_queue.push(message);
+    }
+}
+
+const WAIT_TIME_START = 30000;
+class ConversationManager {
+    constructor() {
+        this.convos = {};
+        this.activeConversation = null;
+        this.awaiting_response = false;
+        this.connection_timeout = null;
+        this.wait_time_limit = WAIT_TIME_START;
+    }
+
+    initAgent(a) {
+        agent = a;
+    }
+
+    _getConvo(name) {
+        if (!this.convos[name])
+            this.convos[name] = new Conversation(name);
+        return this.convos[name];
+    }
+
+    _startMonitor() {
+        clearInterval(this.connection_monitor);
+        let wait_time = 0;
+        let last_time = Date.now();
+        this.connection_monitor = setInterval(() => {
+            if (!this.activeConversation) {
+                this._stopMonitor();
+                return; // will clean itself up
+            }
+
+            let delta = Date.now() - last_time;
+            last_time = Date.now();
+            let convo_partner = this.activeConversation.name;
+
+            if (this.awaiting_response && agent.isIdle()) {
+                wait_time += delta;
+                if (wait_time > this.wait_time_limit) {
+                    agent.handleMessage('system', `${convo_partner} hasn't responded in ${this.wait_time_limit/1000} seconds, respond with a message to them or your own action.`);
+                    wait_time = 0;
+                    this.wait_time_limit*=2;
+                }
+            }
+            else if (!this.awaiting_response){
+                this.wait_time_limit = WAIT_TIME_START;
+                wait_time = 0;
+            }
+
+            if (!this.otherAgentInGame(convo_partner) && !this.connection_timeout) {
+                this.connection_timeout = setTimeout(() => {
+                    if (this.otherAgentInGame(convo_partner)){
+                        this._clearMonitorTimeouts();
+                        return;
+                    }
+                    if (!self_prompter_paused) {
+                        this.endConversation(convo_partner);
+                        agent.handleMessage('system', `${convo_partner} disconnected, conversation has ended.`);
+                    }
+                    else {
+                        this.endConversation(convo_partner);
+                    }
+                }, 10000);
+            }
+        }, 1000);
+    }
+
+    _stopMonitor() {
+        clearInterval(this.connection_monitor);
+        this.connection_monitor = null;
+        this._clearMonitorTimeouts();
+    }
+
+    _clearMonitorTimeouts() {
+        this.awaiting_response = false;
+        clearTimeout(this.connection_timeout);
+        this.connection_timeout = null;
+    }
+
+    async startConversation(send_to, message) {
+        const convo = this._getConvo(send_to);
+        convo.reset();
+        
+        if (agent.self_prompter.on) {
+            await agent.self_prompter.stop();
+            self_prompter_paused = true;
+        }
+        if (convo.active)
+            return;
+        convo.active = true;
+        this.activeConversation = convo;
+        this._startMonitor();
+        this.sendToBot(send_to, message, true, false);
+    }
+
+    startConversationFromOtherBot(name) {
+        const convo = this._getConvo(name);
+        convo.active = true;
+        this.activeConversation = convo;
+        this._startMonitor();
+    }
+
+    sendToBot(send_to, message, start=false, open_chat=true) {
+        if (!this.isOtherAgent(send_to)) {
+            console.warn(`${agent.name} tried to send bot message to non-bot ${send_to}`);
+            return;
+        }
+        const convo = this._getConvo(send_to);
+        
+        if (settings.chat_bot_messages && open_chat)
+            agent.openChat(`(To ${send_to}) ${message}`);
+        
+        if (convo.ignore_until_start)
+            return;
+        convo.active = true;
+        
+        const end = message.includes('!endConversation');
+        const json = {
+            'message': message,
+            start,
+            end,
+        };
+
+        this.awaiting_response = true;
+        sendBotChatToServer(send_to, json);
+    }
+
+    async receiveFromBot(sender, received) {
+        const convo = this._getConvo(sender);
+
+        if (convo.ignore_until_start && !received.start)
+            return;
+
+        // check if any convo is active besides the sender
+        if (this.inConversation() && !this.inConversation(sender)) {
+            this.sendToBot(sender, `I'm talking to someone else, try again later. !endConversation("${sender}")`, false, false);
+            this.endConversation(sender);
+            return;
+        }
+
+        if (received.start) {
+            convo.reset();
+            this.startConversationFromOtherBot(sender);
+        }
+
+        this._clearMonitorTimeouts();
+        convo.queue(received);
+        
+        // responding to conversation takes priority over self prompting
+        if (agent.self_prompter.on){
+            await agent.self_prompter.stopLoop();
+            self_prompter_paused = true;
+        }
+    
+        _scheduleProcessInMessage(sender, received, convo);
+    }
+
+    responseScheduledFor(sender) {
+        if (!this.isOtherAgent(sender) || !this.inConversation(sender))
+            return false;
+        const convo = this._getConvo(sender);
+        return !!convo.inMessageTimer;
+    }
+
+    isOtherAgent(name) {
+        return agent_names.some((n) => n === name);
+    }
+
+    otherAgentInGame(name) {
+        return agents_in_game.some((n) => n === name);
+    }
+    
+    updateAgents(agents) {
+        agent_names = agents.map(a => a.name);
+        agents_in_game = agents.filter(a => a.in_game).map(a => a.name);
+    }
+
+    getInGameAgents() {
+        return agents_in_game;
+    }
+    
+    inConversation(other_agent=null) {
+        if (other_agent)
+            return this.convos[other_agent]?.active;
+        return Object.values(this.convos).some(c => c.active);
+    }
+    
+    endConversation(sender) {
+        if (this.convos[sender]) {
+            this.convos[sender].end();
+            if (this.activeConversation.name === sender) {
+                this._stopMonitor();
+                this.activeConversation = null;
+                if (self_prompter_paused && !this.inConversation()) {
+                    _resumeSelfPrompter();
+                }
+            }
+        }
+    }
+    
+    endAllConversations() {
+        for (const sender in this.convos) {
+            this.endConversation(sender);
+        }
+        if (self_prompter_paused) {
+            _resumeSelfPrompter();
+        }
+    }
+
+    forceEndCurrentConversation() {
+        if (this.activeConversation) {
+            let sender = this.activeConversation.name;
+            this.sendToBot(sender, '!endConversation("' + sender + '")', false, false);
+            this.endConversation(sender);
+        }
+    }
+
+    scheduleSelfPrompter() {
+        self_prompter_paused = true;
+    }
+    
+    cancelSelfPrompter() {
+        self_prompter_paused = false;
+    }
+}
+
+const convoManager = new ConversationManager();
+export default convoManager;
+
+/*
+This function controls conversation flow by deciding when the bot responds.
+The logic is as follows:
+- If neither bot is busy, respond quickly with a small delay.
+- If only the other bot is busy, respond with a long delay to allow it to finish short actions (ex check inventory)
+- If I'm busy but other bot isn't, let LLM decide whether to respond
+- If both bots are busy, don't respond until someone is done, excluding a few actions that allow fast responses
+- New messages received during the delay will reset the delay following this logic, and be queued to respond in bulk
+*/
+const talkOverActions = ['stay', 'followPlayer', 'mode:']; // all mode actions
+const fastDelay = 200;
+const longDelay = 5000;
+async function _scheduleProcessInMessage(sender, received, convo) {
+    if (convo.inMessageTimer)
+        clearTimeout(convo.inMessageTimer);
+    let otherAgentBusy = containsCommand(received.message);
+
+    const scheduleResponse = (delay) => convo.inMessageTimer = setTimeout(() => _processInMessageQueue(sender), delay);
+
+    if (!agent.isIdle() && otherAgentBusy) {
+        // both are busy
+        let canTalkOver = talkOverActions.some(a => agent.actions.currentActionLabel.includes(a));
+        if (canTalkOver)
+            scheduleResponse(fastDelay)
+        // otherwise don't respond
+    }
+    else if (otherAgentBusy)
+        // other bot is busy but I'm not
+        scheduleResponse(longDelay);
+    else if (!agent.isIdle()) {
+        // I'm busy but other bot isn't
+        let canTalkOver = talkOverActions.some(a => agent.actions.currentActionLabel.includes(a));
+        if (canTalkOver) {
+            scheduleResponse(fastDelay);
+        }
+        else {
+            let shouldRespond = await agent.prompter.promptShouldRespondToBot(received.message);
+            console.log(`${agent.name} decided to ${shouldRespond?'respond':'not respond'} to ${sender}`);
+            if (shouldRespond)
+                scheduleResponse(fastDelay);
+        }
+    }
+    else {
+        // neither are busy
+        scheduleResponse(fastDelay);
+    }
+}
+
+function _processInMessageQueue(name) {
+    const convo = convoManager._getConvo(name);
+    _handleFullInMessage(name, _compileInMessages(convo));
+}
+
+function _compileInMessages(convo) {
+    let pack = {};
+    let full_message = '';
+    while (convo.in_queue.length > 0) {
+        pack = convo.in_queue.shift();
+        full_message += pack.message;
+    }
+    pack.message = full_message;
+    return pack;
+}
+
+function _handleFullInMessage(sender, received) {
+    console.log(`${agent.name} responding to "${received.message}" from ${sender}`);
+    
+    const convo = convoManager._getConvo(sender);
+    convo.active = true;
+
+    let message = _tagMessage(received.message);
+    if (received.end) {
+        convoManager.endConversation(sender);
+        message = `Conversation with ${sender} ended with message: "${message}"`;
+        sender = 'system'; // bot will respond to system instead of the other bot
+    }
+    else if (received.start)
+        agent.shut_up = false;
+    convo.inMessageTimer = null;
+    agent.handleMessage(sender, message);
+}
+
+
+function _tagMessage(message) {
+    return "(FROM OTHER BOT)" + message;
+}
+
+async function _resumeSelfPrompter() {
+    await new Promise(resolve => setTimeout(resolve, 5000));
+    if (self_prompter_paused && !convoManager.inConversation()) {
+        self_prompter_paused = false;
+        agent.self_prompter.start();
+    }
+}

src/agent/history.js

@@ -0,0 +1,119 @@
+import { writeFileSync, readFileSync, mkdirSync, existsSync } from 'fs';
+import { NPCData } from './npc/data.js';
+import settings from '../../settings.js';
+
+
+export class History {
+    constructor(agent) {
+        this.agent = agent;
+        this.name = agent.name;
+        this.memory_fp = `./bots/${this.name}/memory.json`;
+        this.full_history_fp = undefined;
+
+        mkdirSync(`./bots/${this.name}/histories`, { recursive: true });
+
+        this.turns = [];
+
+        // Natural language memory as a summary of recent messages + previous memory
+        this.memory = '';
+
+        // Maximum number of messages to keep in context before saving chunk to memory
+        this.max_messages = settings.max_messages;
+
+        // Number of messages to remove from current history and save into memory
+        this.summary_chunk_size = 5; 
+        // chunking reduces expensive calls to promptMemSaving and appendFullHistory
+        // and improves the quality of the memory summary
+    }
+
+    getHistory() { // expects an Examples object
+        return JSON.parse(JSON.stringify(this.turns));
+    }
+
+    async summarizeMemories(turns) {
+        console.log("Storing memories...");
+        this.memory = await this.agent.prompter.promptMemSaving(turns);
+
+        if (this.memory.length > 500) {
+            this.memory = this.memory.slice(0, 500);
+            this.memory += '...(Memory truncated to 500 chars. Compress it more next time)';
+        }
+
+        console.log("Memory updated to: ", this.memory);
+    }
+
+    appendFullHistory(to_store) {
+        if (this.full_history_fp === undefined) {
+            const string_timestamp = new Date().toLocaleString().replace(/[/:]/g, '-').replace(/ /g, '').replace(/,/g, '_');
+            this.full_history_fp = `./bots/${this.name}/histories/${string_timestamp}.json`;
+            writeFileSync(this.full_history_fp, '[]', 'utf8');
+        }
+        try {
+            const data = readFileSync(this.full_history_fp, 'utf8');
+            let full_history = JSON.parse(data);
+            full_history.push(...to_store);
+            writeFileSync(this.full_history_fp, JSON.stringify(full_history, null, 4), 'utf8');
+        } catch (err) {
+            console.error(`Error reading ${this.name}'s full history file: ${err.message}`);
+        }
+    }
+
+    async add(name, content) {
+        let role = 'assistant';
+        if (name === 'system') {
+            role = 'system';
+        }
+        else if (name !== this.name) {
+            role = 'user';
+            content = `${name}: ${content}`;
+        }
+        this.turns.push({role, content});
+
+        if (this.turns.length >= this.max_messages) {
+            let chunk = this.turns.splice(0, this.summary_chunk_size);
+            while (this.turns.length > 0 && this.turns[0].role === 'assistant')
+                chunk.push(this.turns.shift()); // remove until turns starts with system/user message
+
+            await this.summarizeMemories(chunk);
+            this.appendFullHistory(chunk);
+        }
+    }
+
+    async save() {
+        try {
+            const data = {
+                memory: this.memory,
+                turns: this.turns,
+                self_prompt: this.agent.self_prompter.on ? this.agent.self_prompter.prompt : null,
+                last_sender: this.agent.last_sender
+            };
+            writeFileSync(this.memory_fp, JSON.stringify(data, null, 2));
+            console.log('Saved memory to:', this.memory_fp);
+        } catch (error) {
+            console.error('Failed to save history:', error);
+            throw error;
+        }
+    }
+
+    load() {
+        try {
+            if (!existsSync(this.memory_fp)) {
+                console.log('No memory file found.');
+                return null;
+            }
+            const data = JSON.parse(readFileSync(this.memory_fp, 'utf8'));
+            this.memory = data.memory || '';
+            this.turns = data.turns || [];
+            console.log('Loaded memory:', this.memory);
+            return data;
+        } catch (error) {
+            console.error('Failed to load history:', error);
+            throw error;
+        }
+    }
+
+    clear() {
+        this.turns = [];
+        this.memory = '';
+    }
+}

src/agent/library/index.js

@@ -0,0 +1,23 @@
+import * as skills from './skills.js';
+import * as world from './world.js';
+
+
+export function docHelper(functions, module_name) {
+    let docArray = [];
+    for (let skillFunc of functions) {
+        let str = skillFunc.toString();
+        if (str.includes('/**')) {
+            let docEntry = `${module_name}.${skillFunc.name}\n`;
+            docEntry += str.substring(str.indexOf('/**') + 3, str.indexOf('**/')).trim();
+            docArray.push(docEntry);
+        }
+    }
+    return docArray;
+}
+
+export function getSkillDocs() {
+    let docArray = [];
+    docArray = docArray.concat(docHelper(Object.values(skills), 'skills'));
+    docArray = docArray.concat(docHelper(Object.values(world), 'world'));
+    return docArray;
+}

src/agent/library/lockdown.js

@@ -0,0 +1,26 @@
+import 'ses';
+
+// This sets up the secure environment
+// We disable some of the taming to allow for more flexibility
+
+// For configuration, see https://github.com/endojs/endo/blob/master/packages/ses/docs/lockdown.md
+lockdown({
+  // basic devex and quality of life improvements
+  localeTaming: 'unsafe',
+  consoleTaming: 'unsafe',
+  errorTaming: 'unsafe',
+  stackFiltering: 'verbose',
+  // allow eval outside of created compartments
+  // (mineflayer dep "protodef" uses eval)
+  evalTaming: 'unsafeEval',
+});
+
+export const makeCompartment = (endowments = {}) => {
+  return new Compartment({
+    // provide untamed Math, Date, etc
+    Math,
+    Date,
+    // standard endowments
+    ...endowments
+  });
+}

src/agent/library/skill_library.js

@@ -0,0 +1,47 @@
+import { cosineSimilarity } from '../../utils/math.js';
+import { getSkillDocs } from './index.js';
+
+export class SkillLibrary {
+    constructor(agent,embedding_model) {
+        this.agent = agent;
+        this.embedding_model = embedding_model;
+        this.skill_docs_embeddings = {};
+    }
+    async initSkillLibrary() {
+        const skillDocs = getSkillDocs();
+        const embeddingPromises = skillDocs.map((doc) => {
+            return (async () => {
+                let func_name_desc = doc.split('\n').slice(0, 2).join('');
+                this.skill_docs_embeddings[doc] = await this.embedding_model.embed(func_name_desc);
+            })();
+        });
+        await Promise.all(embeddingPromises);
+    }
+
+    async getRelevantSkillDocs(message, select_num) {
+        let latest_message_embedding = '';
+        if(message) //message is not empty, get the relevant skill docs, else return all skill docs
+            latest_message_embedding = await this.embedding_model.embed(message);
+
+        let skill_doc_similarities = Object.keys(this.skill_docs_embeddings)
+            .map(doc_key => ({
+                doc_key,
+                similarity_score: cosineSimilarity(latest_message_embedding, this.skill_docs_embeddings[doc_key])
+            }))
+            .sort((a, b) => b.similarity_score - a.similarity_score);
+
+        let length = skill_doc_similarities.length;
+        if (typeof select_num !== 'number' || isNaN(select_num) || select_num < 0) {
+            select_num = length;
+        } else {
+            select_num = Math.min(Math.floor(select_num), length);
+        }
+        let selected_docs = skill_doc_similarities.slice(0, select_num);
+        let relevant_skill_docs = '#### RELEVENT DOCS INFO ###\nThe following functions are listed in descending order of relevance.\n';
+        relevant_skill_docs += 'SkillDocs:\n'
+        relevant_skill_docs += selected_docs.map(doc => `${doc.doc_key}`).join('\n### ');
+        return relevant_skill_docs;
+    }
+
+
+}

src/agent/library/world.js

@@ -0,0 +1,387 @@
+import pf from 'mineflayer-pathfinder';
+import * as mc from '../../utils/mcdata.js';
+
+
+export function getNearestFreeSpace(bot, size=1, distance=8) {
+    /**
+     * Get the nearest empty space with solid blocks beneath it of the given size.
+     * @param {Bot} bot - The bot to get the nearest free space for.
+     * @param {number} size - The (size x size) of the space to find, default 1.
+     * @param {number} distance - The maximum distance to search, default 8.
+     * @returns {Vec3} - The south west corner position of the nearest free space.
+     * @example
+     * let position = world.getNearestFreeSpace(bot, 1, 8);
+     **/
+    let empty_pos = bot.findBlocks({
+        matching: (block) => {
+            return block && block.name == 'air';
+        },
+        maxDistance: distance,
+        count: 1000
+    });
+    for (let i = 0; i < empty_pos.length; i++) {
+        let empty = true;
+        for (let x = 0; x < size; x++) {
+            for (let z = 0; z < size; z++) {
+                let top = bot.blockAt(empty_pos[i].offset(x, 0, z));
+                let bottom = bot.blockAt(empty_pos[i].offset(x, -1, z));
+                if (!top || !top.name == 'air' || !bottom || bottom.drops.length == 0 || !bottom.diggable) {
+                    empty = false;
+                    break;
+                }
+            }
+            if (!empty) break;
+        }
+        if (empty) {
+            return empty_pos[i];
+        }
+    }
+}
+
+
+export function getBlockAtPosition(bot, x=0, y=0, z=0) {
+     /**
+     * Get a block from the bot's relative position 
+     * @param {Bot} bot - The bot to get the block for.
+     * @param {number} x - The relative x offset to serach, default 0.
+     * @param {number} y - The relative y offset to serach, default 0.
+     * @param {number} y - The relative z offset to serach, default 0. 
+     * @returns {Block} - The nearest block.
+     * @example
+     * let blockBelow = world.getBlockAtPosition(bot, 0, -1, 0);
+     * let blockAbove = world.getBlockAtPosition(bot, 0, 2, 0); since minecraft position is at the feet
+     **/
+    let block = bot.blockAt(bot.entity.position.offset(x, y, z));
+    if (!block) block = {name: 'air'};
+       
+    return block;
+}
+
+
+export function getSurroundingBlocks(bot) {
+    /**
+     * Get the surrounding blocks from the bot's environment.
+     * @param {Bot} bot - The bot to get the block for.
+     * @returns {string[]} - A list of block results as strings.
+     * @example
+     **/
+    // Create a list of block position results that can be unpacked.
+    let res = [];
+    res.push(`Block Below: ${getBlockAtPosition(bot, 0, -1, 0).name}`);
+    res.push(`Block at Legs: ${getBlockAtPosition(bot, 0, 0, 0).name}`);
+    res.push(`Block at Head: ${getBlockAtPosition(bot, 0, 1, 0).name}`);
+
+    return res;
+}
+
+
+export function getFirstBlockAboveHead(bot, ignore_types=null, distance=32) {
+     /**
+     * Searches a column from the bot's position for the first solid block above its head
+     * @param {Bot} bot - The bot to get the block for.
+     * @param {string[]} ignore_types - The names of the blocks to ignore.
+     * @param {number} distance - The maximum distance to search, default 32.
+     * @returns {string} - The fist block above head.
+     * @example
+     * let firstBlockAboveHead = world.getFirstBlockAboveHead(bot, null, 32);
+     **/
+    // if ignore_types is not a list, make it a list.
+    let ignore_blocks = []; 
+    if (ignore_types === null) ignore_blocks = ['air', 'cave_air'];
+    else {
+        if (!Array.isArray(ignore_types))
+            ignore_types = [ignore_types];
+        for(let ignore_type of ignore_types) {
+            if (mc.getBlockId(ignore_type)) ignore_blocks.push(ignore_type);
+        }
+    }
+    // The block above, stops when it finds a solid block .
+    let block_above = {name: 'air'};
+    let height = 0
+    for (let i = 0; i < distance; i++) {
+        let block = bot.blockAt(bot.entity.position.offset(0, i+2, 0));
+        if (!block) block = {name: 'air'};
+        // Ignore and continue
+        if (ignore_blocks.includes(block.name)) continue;
+        // Defaults to any block
+        block_above = block;
+        height = i;
+        break;
+    }
+
+    if (ignore_blocks.includes(block_above.name)) return 'none';
+    
+    return `${block_above.name} (${height} blocks up)`;
+}
+
+
+export function getNearestBlocks(bot, block_types=null, distance=16, count=10000) {
+    /**
+     * Get a list of the nearest blocks of the given types.
+     * @param {Bot} bot - The bot to get the nearest block for.
+     * @param {string[]} block_types - The names of the blocks to search for.
+     * @param {number} distance - The maximum distance to search, default 16.
+     * @param {number} count - The maximum number of blocks to find, default 10000.
+     * @returns {Block[]} - The nearest blocks of the given type.
+     * @example
+     * let woodBlocks = world.getNearestBlocks(bot, ['oak_log', 'birch_log'], 16, 1);
+     **/
+    // if blocktypes is not a list, make it a list
+    let block_ids = [];
+    if (block_types === null) {
+        block_ids = mc.getAllBlockIds(['air']);
+    }
+    else {
+        if (!Array.isArray(block_types))
+            block_types = [block_types];
+        for(let block_type of block_types) {
+            block_ids.push(mc.getBlockId(block_type));
+        }
+    }
+
+    let positions = bot.findBlocks({matching: block_ids, maxDistance: distance, count: count});
+    let blocks = [];
+    for (let i = 0; i < positions.length; i++) {
+        let block = bot.blockAt(positions[i]);
+        let distance = positions[i].distanceTo(bot.entity.position);
+        blocks.push({ block: block, distance: distance });
+    }
+    blocks.sort((a, b) => a.distance - b.distance);
+
+    let res = [];
+    for (let i = 0; i < blocks.length; i++) {
+        res.push(blocks[i].block);
+    }
+    return res;
+}
+
+
+export function getNearestBlock(bot, block_type, distance=16) {
+     /**
+     * Get the nearest block of the given type.
+     * @param {Bot} bot - The bot to get the nearest block for.
+     * @param {string} block_type - The name of the block to search for.
+     * @param {number} distance - The maximum distance to search, default 16.
+     * @returns {Block} - The nearest block of the given type.
+     * @example
+     * let coalBlock = world.getNearestBlock(bot, 'coal_ore', 16);
+     **/
+    let blocks = getNearestBlocks(bot, block_type, distance, 1);
+    if (blocks.length > 0) {
+        return blocks[0];
+    }
+    return null;
+}
+
+
+export function getNearbyEntities(bot, maxDistance=16) {
+    let entities = [];
+    for (const entity of Object.values(bot.entities)) {
+        const distance = entity.position.distanceTo(bot.entity.position);
+        if (distance > maxDistance) continue;
+        entities.push({ entity: entity, distance: distance });
+    }
+    entities.sort((a, b) => a.distance - b.distance);
+    let res = [];
+    for (let i = 0; i < entities.length; i++) {
+        res.push(entities[i].entity);
+    }
+    return res;
+}
+
+export function getNearestEntityWhere(bot, predicate, maxDistance=16) {
+    return bot.nearestEntity(entity => predicate(entity) && bot.entity.position.distanceTo(entity.position) < maxDistance);
+}
+
+
+export function getNearbyPlayers(bot, maxDistance) {
+    if (maxDistance == null) maxDistance = 16;
+    let players = [];
+    for (const entity of Object.values(bot.entities)) {
+        const distance = entity.position.distanceTo(bot.entity.position);
+        if (distance > maxDistance) continue;
+        if (entity.type == 'player' && entity.username != bot.username) {
+            players.push({ entity: entity, distance: distance });
+        } 
+    }
+    players.sort((a, b) => a.distance - b.distance);
+    let res = [];
+    for (let i = 0; i < players.length; i++) {
+        res.push(players[i].entity);
+    }
+    return res;
+}
+
+
+export function getInventoryStacks(bot) {
+    let inventory = [];
+    for (const item of bot.inventory.items()) {
+        if (item != null) {
+            inventory.push(item);
+        }
+    }
+    return inventory;
+}
+
+
+export function getInventoryCounts(bot) {
+    /**
+     * Get an object representing the bot's inventory.
+     * @param {Bot} bot - The bot to get the inventory for.
+     * @returns {object} - An object with item names as keys and counts as values.
+     * @example
+     * let inventory = world.getInventoryCounts(bot);
+     * let oakLogCount = inventory['oak_log'];
+     * let hasWoodenPickaxe = inventory['wooden_pickaxe'] > 0;
+     **/
+    let inventory = {};
+    for (const item of bot.inventory.items()) {
+        if (item != null) {
+            if (inventory[item.name] == null) {
+                inventory[item.name] = 0;
+            }
+            inventory[item.name] += item.count;
+        }
+    }
+    return inventory;
+}
+
+
+export function getCraftableItems(bot) {
+    /**
+     * Get a list of all items that can be crafted with the bot's current inventory.
+     * @param {Bot} bot - The bot to get the craftable items for.
+     * @returns {string[]} - A list of all items that can be crafted.
+     * @example
+     * let craftableItems = world.getCraftableItems(bot);
+     **/
+    let table = getNearestBlock(bot, 'crafting_table');
+    if (!table) {
+        for (const item of bot.inventory.items()) {
+            if (item != null && item.name === 'crafting_table') {
+                table = item;
+                break;
+            }
+        }
+    }
+    let res = [];
+    for (const item of mc.getAllItems()) {
+        let recipes = bot.recipesFor(item.id, null, 1, table);
+        if (recipes.length > 0)
+            res.push(item.name);
+    }
+    return res;
+}
+
+
+export function getPosition(bot) {
+    /**
+     * Get your position in the world (Note that y is vertical).
+     * @param {Bot} bot - The bot to get the position for.
+     * @returns {Vec3} - An object with x, y, and x attributes representing the position of the bot.
+     * @example
+     * let position = world.getPosition(bot);
+     * let x = position.x;
+     **/
+    return bot.entity.position;
+}
+
+
+export function getNearbyEntityTypes(bot) {
+    /**
+     * Get a list of all nearby mob types.
+     * @param {Bot} bot - The bot to get nearby mobs for.
+     * @returns {string[]} - A list of all nearby mobs.
+     * @example
+     * let mobs = world.getNearbyEntityTypes(bot);
+     **/
+    let mobs = getNearbyEntities(bot, 16);
+    let found = [];
+    for (let i = 0; i < mobs.length; i++) {
+        if (!found.includes(mobs[i].name)) {
+            found.push(mobs[i].name);
+        }
+    }
+    return found;
+}
+
+
+export function getNearbyPlayerNames(bot) {
+    /**
+     * Get a list of all nearby player names.
+     * @param {Bot} bot - The bot to get nearby players for.
+     * @returns {string[]} - A list of all nearby players.
+     * @example
+     * let players = world.getNearbyPlayerNames(bot);
+     **/
+    let players = getNearbyPlayers(bot, 64);
+    let found = [];
+    for (let i = 0; i < players.length; i++) {
+        if (!found.includes(players[i].username) && players[i].username != bot.username) {
+            found.push(players[i].username);
+        }
+    }
+    return found;
+}
+
+
+export function getNearbyBlockTypes(bot, distance=16) {
+    /**
+     * Get a list of all nearby block names.
+     * @param {Bot} bot - The bot to get nearby blocks for.
+     * @param {number} distance - The maximum distance to search, default 16.
+     * @returns {string[]} - A list of all nearby blocks.
+     * @example
+     * let blocks = world.getNearbyBlockTypes(bot);
+     **/
+    let blocks = getNearestBlocks(bot, null, distance);
+    let found = [];
+    for (let i = 0; i < blocks.length; i++) {
+        if (!found.includes(blocks[i].name)) {
+            found.push(blocks[i].name);
+        }
+    }
+    return found;
+}
+
+export async function isClearPath(bot, target) {
+    /**
+     * Check if there is a path to the target that requires no digging or placing blocks.
+     * @param {Bot} bot - The bot to get the path for.
+     * @param {Entity} target - The target to path to.
+     * @returns {boolean} - True if there is a clear path, false otherwise.
+     */
+    let movements = new pf.Movements(bot)
+    movements.canDig = false;
+    movements.canPlaceOn = false;
+    let goal = new pf.goals.GoalNear(target.position.x, target.position.y, target.position.z, 1);
+    let path = await bot.pathfinder.getPathTo(movements, goal, 100);
+    return path.status === 'success';
+}
+
+export function shouldPlaceTorch(bot) {
+    if (!bot.modes.isOn('torch_placing') || bot.interrupt_code) return false;
+    const pos = getPosition(bot);
+    // TODO: check light level instead of nearby torches, block.light is broken
+    let nearest_torch = getNearestBlock(bot, 'torch', 6);
+    if (!nearest_torch)
+        nearest_torch = getNearestBlock(bot, 'wall_torch', 6);
+    if (!nearest_torch) {
+        const block = bot.blockAt(pos);
+        let has_torch = bot.inventory.items().find(item => item.name === 'torch');
+        return has_torch && block?.name === 'air';
+    }
+    return false;
+}
+
+export function getBiomeName(bot) {
+    /**
+     * Get the name of the biome the bot is in.
+     * @param {Bot} bot - The bot to get the biome for.
+     * @returns {string} - The name of the biome.
+     * @example
+     * let biome = world.getBiomeName(bot);
+     **/
+    const biomeId = bot.world.getBiome(bot.entity.position);
+    return mc.getAllBiomes()[biomeId].name;
+}

src/agent/memory_bank.js

@@ -0,0 +1,25 @@
+export class MemoryBank {
+	constructor() {
+		this.memory = {};
+	}
+
+	rememberPlace(name, x, y, z) {
+		this.memory[name] = [x, y, z];
+	}
+
+	recallPlace(name) {
+		return this.memory[name];
+	}
+
+	getJson() {
+		return this.memory
+	}
+
+	loadJson(json) {
+		this.memory = json;
+	}
+
+	getKeys() {
+		return Object.keys(this.memory).join(', ')
+	}
+}

src/agent/modes.js

@@ -0,0 +1,414 @@
+import * as skills from './library/skills.js';
+import * as world from './library/world.js';
+import * as mc from '../utils/mcdata.js';
+import settings from '../../settings.js'
+import convoManager from './conversation.js';
+
+async function say(agent, message) {
+    agent.bot.modes.behavior_log += message + '\n';
+    if (agent.shut_up || !settings.narrate_behavior) return;
+    agent.openChat(message);
+}
+
+// a mode is a function that is called every tick to respond immediately to the world
+// it has the following fields:
+// on: whether 'update' is called every tick
+// active: whether an action has been triggered by the mode and hasn't yet finished
+// paused: whether the mode is paused by another action that overrides the behavior (eg followplayer implements its own self defense)
+// update: the function that is called every tick (if on is true)
+// when a mode is active, it will trigger an action to be performed but won't wait for it to return output
+
+// the order of this list matters! first modes will be prioritized
+// while update functions are async, they should *not* be awaited longer than ~100ms as it will block the update loop
+// to perform longer actions, use the execute function which won't block the update loop
+const modes_list = [
+    {
+        name: 'self_preservation',
+        description: 'Respond to drowning, burning, and damage at low health. Interrupts all actions.',
+        interrupts: ['all'],
+        on: true,
+        active: false,
+        fall_blocks: ['sand', 'gravel', 'concrete_powder'], // includes matching substrings like 'sandstone' and 'red_sand'
+        update: async function (agent) {
+            const bot = agent.bot;
+            let block = bot.blockAt(bot.entity.position);
+            let blockAbove = bot.blockAt(bot.entity.position.offset(0, 1, 0));
+            if (!block) block = {name: 'air'}; // hacky fix when blocks are not loaded
+            if (!blockAbove) blockAbove = {name: 'air'};
+            if (blockAbove.name === 'water' || blockAbove.name === 'flowing_water') {
+                // does not call execute so does not interrupt other actions
+                if (!bot.pathfinder.goal) {
+                    bot.setControlState('jump', true);
+                }
+            }
+            else if (this.fall_blocks.some(name => blockAbove.name.includes(name))) {
+                execute(this, agent, async () => {
+                    await skills.moveAway(bot, 2);
+                });
+            }
+            else if (block.name === 'lava' || block.name === 'flowing_lava' || block.name === 'fire' ||
+                blockAbove.name === 'lava' || blockAbove.name === 'flowing_lava' || blockAbove.name === 'fire') {
+                say(agent, 'I\'m on fire!'); // TODO: gets stuck in lava
+                execute(this, agent, async () => {
+                    let nearestWater = world.getNearestBlock(bot, 'water', 20);
+                    if (nearestWater) {
+                        const pos = nearestWater.position;
+                        await skills.goToPosition(bot, pos.x, pos.y, pos.z, 0.2);
+                        say(agent, 'Ahhhh that\'s better!');
+                    }
+                    else {
+                        await skills.moveAway(bot, 5);
+                    }
+                });
+            }
+            else if (Date.now() - bot.lastDamageTime < 3000 && (bot.health < 5 || bot.lastDamageTaken >= bot.health)) {
+                say(agent, 'I\'m dying!');
+                execute(this, agent, async () => {
+                    await skills.moveAway(bot, 20);
+                });
+            }
+            else if (agent.isIdle()) {
+                bot.clearControlStates(); // clear jump if not in danger or doing anything else
+            }
+        }
+    },
+    {
+        name: 'unstuck',
+        description: 'Attempt to get unstuck when in the same place for a while. Interrupts some actions.',
+        interrupts: ['all'],
+        on: true,
+        active: false,
+        prev_location: null,
+        distance: 2,
+        stuck_time: 0,
+        last_time: Date.now(),
+        max_stuck_time: 20,
+        update: async function (agent) {
+            if (agent.isIdle()) { 
+                this.prev_location = null;
+                this.stuck_time = 0;
+                return; // don't get stuck when idle
+            }
+            const bot = agent.bot;
+            if (this.prev_location && this.prev_location.distanceTo(bot.entity.position) < this.distance) {
+                this.stuck_time += (Date.now() - this.last_time) / 1000;
+            }
+            else {
+                this.prev_location = bot.entity.position.clone();
+                this.stuck_time = 0;
+            }
+            if (this.stuck_time > this.max_stuck_time) {
+                say(agent, 'I\'m stuck!');
+                this.stuck_time = 0;
+                execute(this, agent, async () => {
+                    const crashTimeout = setTimeout(() => { agent.cleanKill("Got stuck and couldn't get unstuck") }, 10000);
+                    await skills.moveAway(bot, 5);
+                    clearTimeout(crashTimeout);
+                    say(agent, 'I\'m free.');
+                });
+            }
+            this.last_time = Date.now();
+        }
+    },
+    {
+        name: 'cowardice',
+        description: 'Run away from enemies. Interrupts all actions.',
+        interrupts: ['all'],
+        on: true,
+        active: false,
+        update: async function (agent) {
+            const enemy = world.getNearestEntityWhere(agent.bot, entity => mc.isHostile(entity), 16);
+            if (enemy && await world.isClearPath(agent.bot, enemy)) {
+                say(agent, `Aaa! A ${enemy.name.replace("_", " ")}!`);
+                execute(this, agent, async () => {
+                    await skills.avoidEnemies(agent.bot, 24);
+                });
+            }
+        }
+    },
+    {
+        name: 'self_defense',
+        description: 'Attack nearby enemies. Interrupts all actions.',
+        interrupts: ['all'],
+        on: true,
+        active: false,
+        update: async function (agent) {
+            const enemy = world.getNearestEntityWhere(agent.bot, entity => mc.isHostile(entity), 8);
+            if (enemy && await world.isClearPath(agent.bot, enemy)) {
+                say(agent, `Fighting ${enemy.name}!`);
+                execute(this, agent, async () => {
+                    await skills.defendSelf(agent.bot, 8);
+                });
+            }
+        }
+    },
+    {
+        name: 'hunting',
+        description: 'Hunt nearby animals when idle.',
+        interrupts: [],
+        on: true,
+        active: false,
+        update: async function (agent) {
+            const huntable = world.getNearestEntityWhere(agent.bot, entity => mc.isHuntable(entity), 8);
+            if (huntable && await world.isClearPath(agent.bot, huntable)) {
+                execute(this, agent, async () => {
+                    say(agent, `Hunting ${huntable.name}!`);
+                    await skills.attackEntity(agent.bot, huntable);
+                });
+            }
+        }
+    },
+    {
+        name: 'item_collecting',
+        description: 'Collect nearby items when idle.',
+        interrupts: ['action:followPlayer'],
+        on: true,
+        active: false,
+
+        wait: 2, // number of seconds to wait after noticing an item to pick it up
+        prev_item: null,
+        noticed_at: -1,
+        update: async function (agent) {
+            let item = world.getNearestEntityWhere(agent.bot, entity => entity.name === 'item', 8);
+            let empty_inv_slots = agent.bot.inventory.emptySlotCount();
+            if (item && item !== this.prev_item && await world.isClearPath(agent.bot, item) && empty_inv_slots > 1) {
+                if (this.noticed_at === -1) {
+                    this.noticed_at = Date.now();
+                }
+                if (Date.now() - this.noticed_at > this.wait * 1000) {
+                    say(agent, `Picking up item!`);
+                    this.prev_item = item;
+                    execute(this, agent, async () => {
+                        await skills.pickupNearbyItems(agent.bot);
+                    });
+                    this.noticed_at = -1;
+                }
+            }
+            else {
+                this.noticed_at = -1;
+            }
+        }
+    },
+    {
+        name: 'torch_placing',
+        description: 'Place torches when idle and there are no torches nearby.',
+        interrupts: ['action:followPlayer'],
+        on: true,
+        active: false,
+        cooldown: 5,
+        last_place: Date.now(),
+        update: function (agent) {
+            if (world.shouldPlaceTorch(agent.bot)) {
+                if (Date.now() - this.last_place < this.cooldown * 1000) return;
+                execute(this, agent, async () => {
+                    const pos = agent.bot.entity.position;
+                    await skills.placeBlock(agent.bot, 'torch', pos.x, pos.y, pos.z, 'bottom', true);
+                });
+                this.last_place = Date.now();
+            }
+        }
+    },
+    {
+        name: 'elbow_room',
+        description: 'Move away from nearby players when idle.',
+        interrupts: ['action:followPlayer'],
+        on: true,
+        active: false,
+        distance: 0.5,
+        update: async function (agent) {
+            const player = world.getNearestEntityWhere(agent.bot, entity => entity.type === 'player', this.distance);
+            if (player) {
+                execute(this, agent, async () => {
+                    // wait a random amount of time to avoid identical movements with other bots
+                    const wait_time = Math.random() * 1000;
+                    await new Promise(resolve => setTimeout(resolve, wait_time));
+                    if (player.position.distanceTo(agent.bot.entity.position) < this.distance) {
+                        await skills.moveAway(agent.bot, this.distance);
+                    }
+                });
+            }
+        }
+    },
+    {
+        name: 'idle_staring',
+        description: 'Animation to look around at entities when idle.',
+        interrupts: [],
+        on: true,
+        active: false,
+
+        staring: false,
+        last_entity: null,
+        next_change: 0,
+        update: function (agent) {
+            const entity = agent.bot.nearestEntity();
+            let entity_in_view = entity && entity.position.distanceTo(agent.bot.entity.position) < 10 && entity.name !== 'enderman';
+            if (entity_in_view && entity !== this.last_entity) {
+                this.staring = true;
+                this.last_entity = entity;
+                this.next_change = Date.now() + Math.random() * 1000 + 4000;
+            }
+            if (entity_in_view && this.staring) {
+                let isbaby = entity.type !== 'player' && entity.metadata[16];
+                let height = isbaby ? entity.height/2 : entity.height;
+                agent.bot.lookAt(entity.position.offset(0, height, 0));
+            }
+            if (!entity_in_view)
+                this.last_entity = null;
+            if (Date.now() > this.next_change) {
+                // look in random direction
+                this.staring = Math.random() < 0.3;
+                if (!this.staring) {
+                    const yaw = Math.random() * Math.PI * 2;
+                    const pitch = (Math.random() * Math.PI/2) - Math.PI/4;
+                    agent.bot.look(yaw, pitch, false);
+                }
+                this.next_change = Date.now() + Math.random() * 10000 + 2000;
+            }
+        }
+    },
+    {
+        name: 'cheat',
+        description: 'Use cheats to instantly place blocks and teleport.',
+        interrupts: [],
+        on: false,
+        active: false,
+        update: function (agent) { /* do nothing */ }
+    }
+];
+
+async function execute(mode, agent, func, timeout=-1) {
+    if (agent.self_prompter.on)
+        agent.self_prompter.stopLoop();
+    let interrupted_action = agent.actions.currentActionLabel;
+    mode.active = true;
+    let code_return = await agent.actions.runAction(`mode:${mode.name}`, async () => {
+        await func();
+    }, { timeout });
+    mode.active = false;
+    console.log(`Mode ${mode.name} finished executing, code_return: ${code_return.message}`);
+
+    let should_reprompt = 
+        interrupted_action && // it interrupted a previous action
+        !agent.actions.resume_func && // there is no resume function
+        !agent.self_prompter.on && // self prompting is not on
+        !code_return.interrupted; // this mode action was not interrupted by something else
+
+    if (should_reprompt) {
+        // auto prompt to respond to the interruption
+        let role = convoManager.inConversation() ? agent.last_sender : 'system';
+        let logs = agent.bot.modes.flushBehaviorLog();
+        agent.handleMessage(role, `(AUTO MESSAGE)Your previous action '${interrupted_action}' was interrupted by ${mode.name}.
+        Your behavior log: ${logs}\nRespond accordingly.`);
+    }
+}
+
+let _agent = null;
+const modes_map = {};
+for (let mode of modes_list) {
+    modes_map[mode.name] = mode;
+}
+
+class ModeController {
+    /*
+    SECURITY WARNING:
+    ModesController must be isolated. Do not store references to external objects like `agent`.
+    This object is accessible by LLM generated code, so any stored references are also accessible.
+    This can be used to expose sensitive information by malicious human prompters.
+    */
+    constructor() {
+        this.behavior_log = '';
+    }
+
+    exists(mode_name) {
+        return modes_map[mode_name] != null;
+    }
+
+    setOn(mode_name, on) {
+        modes_map[mode_name].on = on;
+    }
+
+    isOn(mode_name) {
+        return modes_map[mode_name].on;
+    }
+
+    pause(mode_name) {
+        modes_map[mode_name].paused = true;
+    }
+
+    unpause(mode_name) {
+        modes_map[mode_name].paused = false;
+    }
+
+    unPauseAll() {
+        for (let mode of modes_list) {
+            if (mode.paused) console.log(`Unpausing mode ${mode.name}`);
+            mode.paused = false;
+        }
+    }
+
+    getMiniDocs() { // no descriptions
+        let res = 'Agent Modes:';
+        for (let mode of modes_list) {
+            let on = mode.on ? 'ON' : 'OFF';
+            res += `\n- ${mode.name}(${on})`;
+        }
+        return res;
+    }
+
+    getDocs() {
+        let res = 'Agent Modes:';
+        for (let mode of modes_list) {
+            let on = mode.on ? 'ON' : 'OFF';
+            res += `\n- ${mode.name}(${on}): ${mode.description}`;
+        }
+        return res;
+    }
+
+    async update() {
+        if (_agent.isIdle()) {
+            this.unPauseAll();
+        }
+        for (let mode of modes_list) {
+            let interruptible = mode.interrupts.some(i => i === 'all') || mode.interrupts.some(i => i === _agent.actions.currentActionLabel);
+            if (mode.on && !mode.paused && !mode.active && (_agent.isIdle() || interruptible)) {
+                await mode.update(_agent);
+            }
+            if (mode.active) break;
+        }
+    }
+
+    flushBehaviorLog() {
+        const log = this.behavior_log;
+        this.behavior_log = '';
+        return log;
+    }
+
+    getJson() {
+        let res = {};
+        for (let mode of modes_list) {
+            res[mode.name] = mode.on;
+        }
+        return res;
+    }
+
+    loadJson(json) {
+        for (let mode of modes_list) {
+            if (json[mode.name] != undefined) {
+                mode.on = json[mode.name];
+            }
+        }
+    }
+}
+
+export function initModes(agent) {
+    _agent = agent;
+    // the mode controller is added to the bot object so it is accessible from anywhere the bot is used
+    agent.bot.modes = new ModeController();
+    if (agent.task) {
+        agent.bot.restrict_to_inventory = agent.task.restrict_to_inventory;
+    }
+    let modes_json = agent.prompter.getInitModes();
+    if (modes_json) {
+        agent.bot.modes.loadJson(modes_json);
+    }
+}

src/agent/npc/build_goal.js

@@ -0,0 +1,80 @@
+import { Vec3 } from 'vec3';
+import * as skills from '../library/skills.js';
+import * as world from '../library/world.js';
+import * as mc from '../../utils/mcdata.js';
+import { blockSatisfied, getTypeOfGeneric, rotateXZ } from './utils.js';
+
+
+export class BuildGoal {
+    constructor(agent) {
+        this.agent = agent;
+    }
+
+    async wrapSkill(func) {
+        if (!this.agent.isIdle())
+            return false;
+        let res = await this.agent.actions.runAction('BuildGoal', func);
+        return !res.interrupted;
+    }
+
+    async executeNext(goal, position=null, orientation=null) {
+        let sizex = goal.blocks[0][0].length;
+        let sizez = goal.blocks[0].length;
+        let sizey = goal.blocks.length;
+        if (!position) {
+            for (let x = 0; x < sizex - 1; x++) {
+                position = world.getNearestFreeSpace(this.agent.bot, sizex - x, 16);
+                if (position) break;
+            }
+        }
+        if (orientation === null) {
+            orientation = Math.floor(Math.random() * 4);
+        }
+
+        let inventory = world.getInventoryCounts(this.agent.bot);
+        let missing = {};
+        let acted = false;
+        for (let y = goal.offset; y < sizey+goal.offset; y++) {
+            for (let z = 0; z < sizez; z++) {
+                for (let x = 0; x < sizex; x++) {
+
+                    let [rx, rz] = rotateXZ(x, z, orientation, sizex, sizez);
+                    let ry = y - goal.offset;
+                    let block_name = goal.blocks[ry][rz][rx];
+                    if (block_name === null || block_name === '') continue;
+
+                    let world_pos = new Vec3(position.x + x, position.y + y, position.z + z);
+                    let current_block = this.agent.bot.blockAt(world_pos);
+
+                    let res = null;
+                    if (current_block !== null && !blockSatisfied(block_name, current_block)) {
+                        acted = true;
+
+                        if (current_block.name !== 'air') {
+                            res = await this.wrapSkill(async () => {
+                                await skills.breakBlockAt(this.agent.bot, world_pos.x, world_pos.y, world_pos.z);
+                            });
+                            if (!res) return {missing: missing, acted: acted, position: position, orientation: orientation};
+                        }
+
+                        if (block_name !== 'air') {
+                            let block_typed = getTypeOfGeneric(this.agent.bot, block_name);
+                            if (inventory[block_typed] > 0) {
+                                res = await this.wrapSkill(async () => {
+                                    await skills.placeBlock(this.agent.bot, block_typed, world_pos.x, world_pos.y, world_pos.z);
+                                });
+                                if (!res) return {missing: missing, acted: acted, position: position, orientation: orientation};
+                            } else {
+                                if (missing[block_typed] === undefined)
+                                    missing[block_typed] = 0;
+                                missing[block_typed]++;
+                            }
+                        }
+                    }
+                }
+            }
+        }
+        return {missing: missing, acted: acted, position: position, orientation: orientation};
+    }
+
+}

src/agent/npc/construction/dirt_shelter.json

@@ -0,0 +1,38 @@
+{
+    "name": "dirt_shelter",
+    "offset": -2,
+    "blocks": [
+        [
+            ["", "", "", "", ""],
+            ["", "dirt", "dirt", "dirt", ""],
+            ["", "dirt", "dirt", "dirt", ""],
+            ["", "dirt", "dirt", "dirt", ""],
+            ["", "", "dirt", "", ""],
+            ["", "", "dirt", "", ""]
+        ],
+        [
+            ["dirt", "dirt", "dirt", "dirt", "dirt"],
+            ["dirt", "chest", "bed", "air", "dirt"],
+            ["dirt", "air", "bed", "air", "dirt"],
+            ["dirt", "air", "air", "air", "dirt"],
+            ["dirt", "dirt", "door", "dirt", "dirt"],
+            ["dirt", "dirt", "air", "dirt", "dirt"]
+        ],
+        [
+            ["dirt", "dirt", "dirt", "dirt", "dirt"],
+            ["dirt", "air", "air", "air", "dirt"],
+            ["dirt", "torch", "air", "air", "dirt"],
+            ["dirt", "air", "air", "air", "dirt"],
+            ["dirt", "dirt", "door", "dirt", "dirt"],
+            ["air", "air", "air", "air", "air"]
+        ],
+        [
+            ["air", "air", "air", "air", "air"],
+            ["dirt", "dirt", "dirt", "dirt", "dirt"],
+            ["dirt", "dirt", "dirt", "dirt", "dirt"],
+            ["dirt", "dirt", "dirt", "dirt", "dirt"],
+            ["air", "air", "air", "air", "air"],
+            ["air", "air", "air", "air", "air"]
+        ]
+    ]
+}

src/agent/npc/construction/large_house.json

@@ -0,0 +1,230 @@
+{
+    "name": "large_house",
+    "offset": -4,
+    "blocks": [
+        [
+            ["", "", "", "", "", "", "", "", "", "", ""],
+            ["", "", "", "", "", "", "", "", "", "", ""],
+            ["", "", "", "", "", "", "", "", "", "", ""],
+            ["", "cobblestone", "cobblestone", "cobblestone", "cobblestone", "cobblestone", "cobblestone", "cobblestone", "cobblestone", "cobblestone", ""],
+            ["", "cobblestone", "cobblestone", "cobblestone", "cobblestone", "cobblestone", "cobblestone", "cobblestone", "cobblestone", "cobblestone", ""],
+            ["", "cobblestone", "cobblestone", "cobblestone", "cobblestone", "cobblestone", "cobblestone", "cobblestone", "cobblestone", "cobblestone", ""],
+            ["", "cobblestone", "cobblestone", "cobblestone", "cobblestone", "cobblestone", "cobblestone", "cobblestone", "cobblestone", "cobblestone", ""],
+            ["", "cobblestone", "cobblestone", "cobblestone", "cobblestone", "cobblestone", "cobblestone", "cobblestone", "cobblestone", "cobblestone", ""],
+            ["", "cobblestone", "cobblestone", "cobblestone", "cobblestone", "cobblestone", "cobblestone", "cobblestone", "cobblestone", "cobblestone", ""],
+            ["", "cobblestone", "cobblestone", "cobblestone", "cobblestone", "cobblestone", "cobblestone", "cobblestone", "cobblestone", "cobblestone", ""],
+            ["", "", "", "", "", "", "", "", "", "", ""],
+            ["", "", "", "", "", "", "", "", "", "", ""],
+            ["", "", "", "", "", "", "", "", "", "", ""],
+            ["", "", "", "", "", "", "", "", "", "", ""]
+        ],
+        [
+            ["", "", "", "", "", "", "", "", "", "", ""],
+            ["", "", "", "", "", "", "", "", "", "", ""],
+            ["", "", "", "", "", "", "", "", "", "", ""],
+            ["", "cobblestone", "cobblestone", "cobblestone", "cobblestone", "cobblestone", "cobblestone", "cobblestone", "cobblestone", "cobblestone", ""],
+            ["", "cobblestone", "air", "air", "air", "air", "air", "air", "air", "cobblestone", ""],
+            ["", "cobblestone", "air", "air", "air", "air", "air", "air", "air", "cobblestone", ""],
+            ["", "cobblestone", "air", "air", "air", "air", "air", "air", "air", "cobblestone", ""],
+            ["", "cobblestone", "planks", "air", "air", "air", "air", "air", "air", "cobblestone", ""],
+            ["", "cobblestone", "planks", "air", "air", "air", "air", "air", "air", "cobblestone", ""],
+            ["", "cobblestone", "cobblestone", "cobblestone", "cobblestone", "cobblestone", "cobblestone", "cobblestone", "cobblestone", "cobblestone", ""],
+            ["", "", "", "", "", "", "", "", "", "", ""],
+            ["", "", "", "", "", "", "", "", "", "", ""],
+            ["", "", "", "", "", "", "", "", "", "", ""],
+            ["", "", "", "", "", "", "", "", "", "", ""]
+        ],
+        [
+            ["", "", "", "", "", "", "", "", "", "", ""],
+            ["", "", "", "", "", "", "", "", "", "", ""],
+            ["", "", "", "", "", "", "", "", "", "", ""],
+            ["", "cobblestone", "cobblestone", "cobblestone", "cobblestone", "cobblestone", "cobblestone", "cobblestone", "cobblestone", "cobblestone", ""],
+            ["", "cobblestone", "air", "torch", "air", "air", "air", "torch", "air", "cobblestone", ""],
+            ["", "cobblestone", "air", "air", "air", "air", "air", "air", "air", "cobblestone", ""],
+            ["", "cobblestone", "air", "air", "air", "air", "air", "air", "air", "cobblestone", ""],
+            ["", "cobblestone", "air", "air", "air", "air", "air", "air", "air", "cobblestone", ""],
+            ["", "cobblestone", "planks", "torch", "air", "air", "air", "torch", "air", "cobblestone", ""],
+            ["", "cobblestone", "cobblestone", "cobblestone", "cobblestone", "cobblestone", "cobblestone", "cobblestone", "cobblestone", "cobblestone", ""],
+            ["", "", "", "", "", "", "", "", "", "", ""],
+            ["", "", "", "", "", "", "", "", "", "", ""],
+            ["", "", "", "", "", "", "", "", "", "", ""],
+            ["", "", "", "", "", "", "", "", "", "", ""]
+        ],
+        [
+            ["", "", "", "cobblestone", "cobblestone", "cobblestone", "cobblestone", "cobblestone", "", "", ""],
+            ["", "", "", "cobblestone", "cobblestone", "cobblestone", "cobblestone", "cobblestone", "", "", ""],
+            ["", "", "", "cobblestone", "cobblestone", "cobblestone", "cobblestone", "cobblestone", "", "", ""],
+            ["cobblestone", "cobblestone", "cobblestone", "cobblestone", "cobblestone", "cobblestone", "cobblestone", "cobblestone", "cobblestone", "cobblestone", ""],
+            ["cobblestone", "cobblestone", "cobblestone", "cobblestone", "cobblestone", "cobblestone", "cobblestone", "cobblestone", "cobblestone", "cobblestone", ""],
+            ["cobblestone", "cobblestone", "cobblestone", "cobblestone", "cobblestone", "cobblestone", "cobblestone", "cobblestone", "cobblestone", "cobblestone", ""],
+            ["cobblestone", "cobblestone", "cobblestone", "cobblestone", "cobblestone", "cobblestone", "cobblestone", "cobblestone", "cobblestone", "cobblestone", ""],
+            ["cobblestone", "cobblestone", "air", "cobblestone", "cobblestone", "cobblestone", "cobblestone", "cobblestone", "cobblestone", "cobblestone", "dirt"],
+            ["cobblestone", "cobblestone", "air", "cobblestone", "cobblestone", "cobblestone", "cobblestone", "cobblestone", "cobblestone", "cobblestone", ""],
+            ["cobblestone", "cobblestone", "air", "cobblestone", "cobblestone", "cobblestone", "cobblestone", "cobblestone", "cobblestone", "cobblestone", ""],
+            ["", "", "", "cobblestone", "cobblestone", "cobblestone", "cobblestone", "cobblestone", "cobblestone", "cobblestone", ""],
+            ["", "", "", "cobblestone", "cobblestone", "cobblestone", "cobblestone", "cobblestone", "cobblestone", "cobblestone", ""],
+            ["", "", "", "cobblestone", "cobblestone", "cobblestone", "cobblestone", "cobblestone", "cobblestone", "cobblestone", ""],
+            ["", "", "", "cobblestone", "cobblestone", "cobblestone", "cobblestone", "cobblestone", "cobblestone", "cobblestone", ""]
+        ],
+        [
+            ["", "", "", "log", "planks", "planks", "planks", "log", "", "", ""],
+            ["", "", "", "planks", "furnace", "air", "crafting_table", "planks", "", "", ""],
+            ["", "", "", "planks", "air", "air", "air", "planks", "", "", ""],
+            ["log", "planks", "planks", "log", "planks", "air", "planks", "log", "planks", "log", ""],
+            ["planks", "planks", "air", "air", "air", "air", "air", "air", "air", "planks", ""],
+            ["planks", "planks", "air", "air", "air", "air", "air", "air", "air", "planks", ""],
+            ["planks", "planks", "air", "air", "air", "air", "air", "air", "air", "door", "air"],
+            ["planks", "planks", "air", "air", "air", "air", "air", "air", "air", "planks", ""],
+            ["planks", "planks", "air", "air", "air", "air", "air", "air", "air", "planks", ""],
+            ["log", "planks", "planks", "log", "planks", "planks", "air", "planks", "planks", "log", ""],
+            ["", "", "", "planks", "air", "air", "air", "", "air", "planks", ""],
+            ["", "", "", "planks", "chest", "air", "air", "bed", "", "planks", ""],
+            ["", "", "", "planks", "chest", "air", "air", "", "air", "planks", ""],
+            ["", "", "", "log", "planks", "planks", "planks", "planks", "planks", "log", ""]
+        ],
+        [
+            ["", "", "", "log", "planks", "planks", "planks", "log", "", "", ""],
+            ["", "", "", "planks", "air", "air", "air", "glass", "", "", ""],
+            ["", "", "", "planks", "air", "air", "air", "glass", "", "", ""],
+            ["log", "planks", "planks", "log", "planks", "air", "planks", "log", "planks", "log", ""],
+            ["planks", "air", "air", "air", "air", "air", "air", "air", "air", "planks", ""],
+            ["planks", "planks", "air", "air", "air", "air", "air", "air", "air", "planks", ""],
+            ["planks", "planks", "air", "air", "air", "air", "air", "air", "air", "door", "air"],
+            ["planks", "planks", "air", "air", "air", "air", "air", "air", "air", "planks", ""],
+            ["planks", "planks", "air", "air", "air", "air", "air", "air", "air", "planks", ""],
+            ["log", "planks", "planks", "log", "planks", "planks", "air", "planks", "planks", "log", ""],
+            ["", "", "", "planks", "air", "air", "air", "air", "air", "planks", ""],
+            ["", "", "", "planks", "air", "air", "air", "air", "air", "planks", ""],
+            ["", "", "", "planks", "air", "air", "air", "air", "air", "planks", ""],
+            ["", "", "", "log", "planks", "glass", "glass", "glass", "planks", "log", ""]
+        ],
+        [
+            ["", "", "", "log", "planks", "planks", "planks", "log", "", "", ""],
+            ["", "", "", "planks", "air", "air", "air", "glass", "", "", ""],
+            ["", "", "", "planks", "torch", "air", "torch", "glass", "", "", ""],
+            ["log", "planks", "planks", "log", "planks", "air", "planks", "log", "planks", "log", ""],
+            ["planks", "air", "air", "torch", "air", "air", "air", "air", "air", "planks", ""],
+            ["planks", "air", "air", "air", "air", "air", "air", "air", "torch", "planks", ""],
+            ["planks", "planks", "air", "air", "air", "air", "air", "air", "air", "planks", ""],
+            ["planks", "planks", "air", "air", "air", "air", "air", "air", "torch", "planks", ""],
+            ["planks", "planks", "air", "torch", "air", "air", "air", "air", "air", "planks", ""],
+            ["log", "planks", "planks", "log", "planks", "planks", "air", "planks", "planks", "log", ""],
+            ["", "", "", "planks", "air", "torch", "air", "torch", "air", "planks", ""],
+            ["", "", "", "planks", "air", "air", "air", "air", "air", "planks", ""],
+            ["", "", "", "planks", "air", "air", "air", "air", "air", "planks", ""],
+            ["", "", "", "log", "planks", "glass", "glass", "glass", "planks", "log", ""]
+        ],
+        [
+            ["", "", "", "log", "log", "log", "log", "log", "", "", ""],
+            ["", "", "", "log", "planks", "planks", "planks", "log", "", "", ""],
+            ["", "", "", "log", "planks", "planks", "planks", "log", "", "", ""],
+            ["log", "log", "log", "log", "log", "log", "log", "log", "log", "log", ""],
+            ["log", "air", "planks", "planks", "planks", "planks", "planks", "planks", "planks", "log", ""],
+            ["log", "air", "planks", "planks", "planks", "planks", "planks", "planks", "planks", "log", ""],
+            ["log", "air", "planks", "planks", "planks", "planks", "planks", "planks", "planks", "log", ""],
+            ["log", "planks", "planks", "planks", "planks", "planks", "planks", "planks", "planks", "log", ""],
+            ["log", "planks", "planks", "planks", "planks", "planks", "planks", "planks", "planks", "log", ""],
+            ["log", "log", "log", "log", "log", "log", "log", "log", "log", "log", ""],
+            ["", "", "", "log", "planks", "planks", "planks", "planks", "planks", "log", ""],
+            ["", "", "", "log", "planks", "planks", "planks", "planks", "planks", "log", ""],
+            ["", "", "", "log", "planks", "planks", "planks", "planks", "planks", "log", ""],
+            ["", "", "", "log", "log", "log", "log", "log", "log", "log", ""]
+        ],
+        [
+            ["", "", "", "", "", "", "", "", "", "", ""],
+            ["", "", "", "", "planks", "planks", "planks", "", "", "", ""],
+            ["", "", "", "", "planks", "planks", "planks", "", "", "", ""],
+            ["log", "planks", "planks", "log", "planks", "planks", "planks", "planks", "planks", "log", ""],
+            ["planks", "air", "bookshelf", "bookshelf", "air", "air", "air", "air", "torch", "planks", ""],
+            ["planks", "air", "air", "air", "air", "air", "air", "air", "air", "planks", ""],
+            ["planks", "air", "air", "air", "air", "air", "air", "air", "air", "planks", ""],
+            ["planks", "air", "air", "air", "air", "air", "air", "air", "air", "planks", ""],
+            ["planks", "air", "air", "air", "air", "air", "air", "air", "torch", "planks", ""],
+            ["log", "planks", "planks", "log", "planks", "planks", "planks", "planks", "planks", "log", ""],
+            ["", "", "", "", "planks", "planks", "planks", "planks", "planks", "", ""],
+            ["", "", "", "", "planks", "planks", "planks", "planks", "planks", "", ""],
+            ["", "", "", "", "planks", "planks", "planks", "planks", "planks", "", ""],
+            ["", "", "", "", "", "", "", "", "", "", ""]
+        ],
+        [
+            ["", "", "", "", "", "", "", "", "", "", ""],
+            ["", "", "", "", "", "", "", "", "", "", ""],
+            ["", "", "", "", "", "", "", "", "", "", ""],
+            ["log", "planks", "planks", "log", "glass", "glass", "glass", "glass", "glass", "log", ""],
+            ["glass", "air", "bookshelf", "bookshelf", "air", "air", "air", "air", "air", "planks", ""],
+            ["glass", "air", "air", "air", "air", "air", "air", "air", "air", "glass", ""],
+            ["glass", "air", "air", "air", "air", "air", "air", "air", "air", "glass", ""],
+            ["glass", "air", "air", "air", "air", "air", "air", "air", "air", "glass", ""],
+            ["glass", "air", "air", "air", "air", "air", "air", "air", "air", "glass", ""],
+            ["log", "planks", "planks", "log", "glass", "glass", "glass", "glass", "glass", "log", ""],
+            ["", "", "", "", "", "", "", "", "", "", ""],
+            ["", "", "", "", "", "", "", "", "", "", ""],
+            ["", "", "", "", "", "", "", "", "", "", ""],
+            ["", "", "", "", "", "", "", "", "", "", ""]
+        ],
+        [
+            ["", "", "", "", "", "", "", "", "", "", ""],
+            ["", "", "", "", "", "", "", "", "", "", ""],
+            ["", "", "", "", "", "", "", "", "", "", ""],
+            ["log", "planks", "planks", "log", "glass", "glass", "glass", "glass", "glass", "log", ""],
+            ["glass", "air", "air", "torch", "air", "air", "air", "air", "air", "glass", ""],
+            ["glass", "air", "air", "air", "air", "air", "air", "air", "air", "glass", ""],
+            ["glass", "air", "air", "air", "air", "air", "air", "air", "air", "glass", ""],
+            ["glass", "air", "air", "air", "air", "air", "air", "air", "air", "glass", ""],
+            ["glass", "air", "air", "torch", "air", "air", "air", "air", "air", "glass", ""],
+            ["log", "planks", "planks", "log", "glass", "glass", "glass", "glass", "glass", "log", ""],
+            ["", "", "", "", "", "", "", "", "", "", ""],
+            ["", "", "", "", "", "", "", "", "", "", ""],
+            ["", "", "", "", "", "", "", "", "", "", ""],
+            ["", "", "", "", "", "", "", "", "", "", ""]
+        ],
+        [
+            ["", "", "", "", "", "", "", "", "", "", ""],
+            ["", "", "", "", "", "", "", "", "", "", ""],
+            ["", "", "", "", "", "", "", "", "", "", ""],
+            ["log", "log", "log", "log", "log", "log", "log", "log", "log", "log", ""],
+            ["log", "planks", "planks", "log", "planks", "planks", "planks", "planks", "planks", "log", ""],
+            ["log", "planks", "planks", "log", "planks", "planks", "planks", "planks", "planks", "log", ""],
+            ["log", "planks", "planks", "log", "planks", "planks", "planks", "planks", "planks", "log", ""],
+            ["log", "planks", "planks", "log", "planks", "planks", "planks", "planks", "planks", "log", ""],
+            ["log", "planks", "planks", "log", "planks", "planks", "planks", "planks", "planks", "log", ""],
+            ["log", "log", "log", "log", "log", "log", "log", "log", "log", "log", ""],
+            ["", "", "", "", "", "", "", "", "", "", ""],
+            ["", "", "", "", "", "", "", "", "", "", ""],
+            ["", "", "", "", "", "", "", "", "", "", ""],
+            ["", "", "", "", "", "", "", "", "", "", ""]
+        ],
+        [
+            ["", "", "", "", "", "", "", "", "", "", ""],
+            ["", "", "", "", "", "", "", "", "", "", ""],
+            ["", "", "", "", "", "", "", "", "", "", ""],
+            ["", "", "", "", "", "", "", "", "", "", ""],
+            ["", "", "", "", "planks", "planks", "planks", "planks", "planks", "", ""],
+            ["", "", "", "", "planks", "planks", "planks", "planks", "planks", "", ""],
+            ["", "", "", "", "planks", "planks", "planks", "planks", "planks", "", ""],
+            ["", "", "", "", "planks", "planks", "planks", "planks", "planks", "", ""],
+            ["", "", "", "", "planks", "planks", "planks", "planks", "planks", "", ""],
+            ["", "", "", "", "", "", "", "", "", "", ""],
+            ["", "", "", "", "", "", "", "", "", "", ""],
+            ["", "", "", "", "", "", "", "", "", "", ""],
+            ["", "", "", "", "", "", "", "", "", "", ""],
+            ["", "", "", "", "", "", "", "", "", "", ""]
+        ],
+        [
+            ["", "", "", "", "", "", "", "", "", "", ""],
+            ["", "", "", "", "", "", "", "", "", "", ""],
+            ["", "", "", "", "", "", "", "", "", "", ""],
+            ["", "", "", "", "", "", "", "", "", "", ""],
+            ["", "", "", "", "", "", "", "", "", "", ""],
+            ["", "", "", "", "", "planks", "planks", "planks", "", "", ""],
+            ["", "", "", "", "", "planks", "planks", "planks", "", "", ""],
+            ["", "", "", "", "", "planks", "planks", "planks", "", "", ""],
+            ["", "", "", "", "", "", "", "", "", "", ""],
+            ["", "", "", "", "", "", "", "", "", "", ""],
+            ["", "", "", "", "", "", "", "", "", "", ""],
+            ["", "", "", "", "", "", "", "", "", "", ""],
+            ["", "", "", "", "", "", "", "", "", "", ""],
+            ["", "", "", "", "", "", "", "", "", "", ""]
+        ]
+    ]
+}

src/agent/npc/construction/small_stone_house.json

@@ -0,0 +1,42 @@
+{
+    "name": "small_stone_house",
+    "offset": -1,
+    "blocks": [
+        [
+            ["", "", "", "", ""],
+            ["", "planks", "planks", "planks", ""],
+            ["", "planks", "planks", "planks", ""],
+            ["", "planks", "planks", "planks", ""],
+            ["", "planks", "planks", "planks", ""],
+            ["", "", "planks", "", ""],
+            ["", "", "", "", ""]
+        ],
+        [
+            ["cobblestone", "cobblestone", "cobblestone", "cobblestone", "cobblestone"],
+            ["cobblestone", "chest", "bed", "air", "cobblestone"],
+            ["cobblestone", "air", "bed", "air", "cobblestone"],
+            ["cobblestone", "air", "air", "air", "cobblestone"],
+            ["cobblestone", "air", "air", "air", "cobblestone"],
+            ["cobblestone", "cobblestone", "door", "cobblestone", "cobblestone"],
+            ["", "air", "air", "air", ""]
+        ],
+        [
+            ["cobblestone", "cobblestone", "cobblestone", "cobblestone", "cobblestone"],
+            ["cobblestone", "torch", "air", "torch", "cobblestone"],
+            ["cobblestone", "air", "air", "air", "cobblestone"],
+            ["cobblestone", "air", "air", "air", "cobblestone"],
+            ["cobblestone", "torch", "air", "torch", "cobblestone"],
+            ["cobblestone", "cobblestone", "door", "cobblestone", "cobblestone"],
+            ["", "air", "air", "air", ""]
+        ],
+        [
+            ["air", "air", "air", "air", "air"],
+            ["air", "cobblestone", "cobblestone", "cobblestone", "air"],
+            ["cobblestone", "cobblestone", "cobblestone", "cobblestone", "cobblestone"],
+            ["cobblestone", "cobblestone", "cobblestone", "cobblestone", "cobblestone"],
+            ["air", "cobblestone", "cobblestone", "cobblestone", "air"],
+            ["air", "air", "air", "air", "air"],
+            ["", "air", "air", "air", ""]
+        ]
+    ]
+}

src/agent/npc/construction/small_wood_house.json

@@ -0,0 +1,42 @@
+{
+    "name": "small_wood_house",
+    "offset": -1,
+    "blocks": [
+        [
+            ["", "", "", "", ""],
+            ["", "planks", "planks", "planks", ""],
+            ["", "planks", "planks", "planks", ""],
+            ["", "planks", "planks", "planks", ""],
+            ["", "planks", "planks", "planks", ""],
+            ["", "", "planks", "", ""],
+            ["", "", "", "", ""]
+        ],
+        [
+            ["log", "planks", "planks", "planks", "log"],
+            ["planks", "chest", "bed", "air", "planks"],
+            ["planks", "air", "bed", "air", "planks"],
+            ["planks", "air", "air", "air", "planks"],
+            ["planks", "air", "air", "air", "planks"],
+            ["log", "planks", "door", "planks", "log"],
+            ["", "air", "air", "air", ""]
+        ],
+        [
+            ["log", "planks", "planks", "planks", "log"],
+            ["planks", "torch", "air", "torch", "planks"],
+            ["planks", "air", "air", "air", "planks"],
+            ["planks", "air", "air", "air", "planks"],
+            ["planks", "torch", "air", "torch", "planks"],
+            ["log", "planks", "door", "planks", "log"],
+            ["", "air", "air", "air", ""]
+        ],
+        [
+            ["air", "air", "air", "air", "air"],
+            ["air", "planks", "planks", "planks", "air"],
+            ["planks", "planks", "planks", "planks", "planks"],
+            ["planks", "planks", "planks", "planks", "planks"],
+            ["air", "planks", "planks", "planks", "air"],
+            ["air", "air", "air", "air", "air"],
+            ["", "air", "air", "air", ""]
+        ]
+    ]
+}

src/agent/npc/controller.js

@@ -0,0 +1,261 @@
+import { readdirSync, readFileSync } from 'fs';
+import { NPCData } from './data.js';
+import { ItemGoal } from './item_goal.js';
+import { BuildGoal } from './build_goal.js';
+import { itemSatisfied, rotateXZ } from './utils.js';
+import * as skills from '../library/skills.js';
+import * as world from '../library/world.js';
+import * as mc from '../../utils/mcdata.js';
+
+
+export class NPCContoller {
+    constructor(agent) {
+        this.agent = agent;
+        this.data = NPCData.fromObject(agent.prompter.profile.npc);
+        this.temp_goals = [];
+        this.item_goal = new ItemGoal(agent, this.data);
+        this.build_goal = new BuildGoal(agent);
+        this.constructions = {};
+        this.last_goals = {};
+    }
+
+    getBuiltPositions() {
+        let positions = [];
+        for (let name in this.data.built) {
+            let position = this.data.built[name].position;
+            let offset = this.constructions[name].offset;
+            let sizex = this.constructions[name].blocks[0][0].length;
+            let sizez = this.constructions[name].blocks[0].length;
+            let sizey = this.constructions[name].blocks.length;
+            for (let y = offset; y < sizey+offset; y++) {
+                for (let z = 0; z < sizez; z++) {
+                    for (let x = 0; x < sizex; x++) {
+                        positions.push({x: position.x + x, y: position.y + y, z: position.z + z});
+                    }
+                }
+            }
+        }
+        return positions;
+    }
+
+    init() {
+        try {
+            for (let file of readdirSync('src/agent/npc/construction')) {
+                if (file.endsWith('.json')) {
+                    this.constructions[file.slice(0, -5)] = JSON.parse(readFileSync('src/agent/npc/construction/' + file, 'utf8'));
+                }
+            }
+        } catch (e) {
+            console.log('Error reading construction file');
+        }
+
+        for (let name in this.constructions) {
+            let sizez = this.constructions[name].blocks[0].length;
+            let sizex = this.constructions[name].blocks[0][0].length;
+            let max_size = Math.max(sizex, sizez);
+            for (let y = 0; y < this.constructions[name].blocks.length; y++) {
+                for (let z = 0; z < max_size; z++) {
+                    if (z >= this.constructions[name].blocks[y].length)
+                        this.constructions[name].blocks[y].push([]);
+                    for (let x = 0; x < max_size; x++) {
+                        if (x >= this.constructions[name].blocks[y][z].length)
+                            this.constructions[name].blocks[y][z].push('');
+                    }
+                }
+            }
+        }
+
+        this.agent.bot.on('idle', async () => {
+            if (this.data.goals.length === 0 && !this.data.curr_goal) return;
+            // Wait a while for inputs before acting independently
+            await new Promise((resolve) => setTimeout(resolve, 5000));
+            if (!this.agent.isIdle()) return;
+
+            // Persue goal
+            if (!this.agent.actions.resume_func) {
+                this.executeNext();
+                this.agent.history.save();
+            }
+        });
+    }
+
+    async setGoal(name=null, quantity=1) {
+        this.data.curr_goal = null;
+        this.last_goals = {};
+        if (name) {
+            this.data.curr_goal = {name: name, quantity: quantity};
+            return;
+        }
+
+        if (!this.data.do_set_goal) return;
+
+        let past_goals = {...this.last_goals};
+        for (let goal in this.data.goals) {
+            if (past_goals[goal.name] === undefined) past_goals[goal.name] = true;
+        }
+        let res = await this.agent.prompter.promptGoalSetting(this.agent.history.getHistory(), past_goals);
+        if (res) {
+            this.data.curr_goal = res;
+            console.log('Set new goal: ', res.name, ' x', res.quantity);
+        } else {
+            console.log('Error setting new goal.');
+        }
+    }
+
+    async executeNext() {
+        if (!this.agent.isIdle()) return;
+        await this.agent.actions.runAction('npc:moveAway', async () => {
+            await skills.moveAway(this.agent.bot, 2);
+        });
+
+        if (!this.data.do_routine || this.agent.bot.time.timeOfDay < 13000) { 
+            // Exit any buildings
+            let building = this.currentBuilding();
+            if (building == this.data.home) {
+                let door_pos = this.getBuildingDoor(building);
+                if (door_pos) {
+                    await this.agent.actions.runAction('npc:exitBuilding', async () => {
+                        await skills.useDoor(this.agent.bot, door_pos);
+                        await skills.moveAway(this.agent.bot, 2); // If the bot is too close to the building it will try to enter again
+                    });
+                }
+            }
+
+            // Work towards goals
+            await this.executeGoal();
+
+        } else {
+            // Reset goal at the end of the day
+            this.data.curr_goal = null;
+
+            // Return to home
+            let building = this.currentBuilding();
+            if (this.data.home !== null && (building === null || building != this.data.home)) {
+                let door_pos = this.getBuildingDoor(this.data.home);
+                await this.agent.actions.runAction('npc:returnHome', async () => {
+                    await skills.useDoor(this.agent.bot, door_pos);
+                });
+            }
+
+            // Go to bed
+            await this.agent.actions.runAction('npc:bed', async () => {
+                await skills.goToBed(this.agent.bot);
+            });
+        }
+
+        if (this.agent.isIdle())
+            this.agent.bot.emit('idle');
+    }
+
+    async executeGoal() {
+        // If we need more blocks to complete a building, get those first
+        let goals = this.temp_goals.concat(this.data.goals);
+        if (this.data.curr_goal)
+            goals = goals.concat([this.data.curr_goal])
+        this.temp_goals = [];
+
+        let acted = false;
+        for (let goal of goals) {
+
+            // Obtain goal item or block
+            if (this.constructions[goal.name] === undefined) {
+                if (!itemSatisfied(this.agent.bot, goal.name, goal.quantity)) {
+                    let res = await this.item_goal.executeNext(goal.name, goal.quantity);
+                    this.last_goals[goal.name] = res;
+                    acted = true;
+                    break;
+                }
+            }
+
+            // Build construction goal
+            else {
+                let res = null;
+                if (this.data.built.hasOwnProperty(goal.name)) {
+                    res = await this.build_goal.executeNext(
+                        this.constructions[goal.name],
+                        this.data.built[goal.name].position,
+                        this.data.built[goal.name].orientation
+                    );
+                } else {
+                    res = await this.build_goal.executeNext(this.constructions[goal.name]);
+                    this.data.built[goal.name] = {
+                        name: goal.name,
+                        position: res.position,
+                        orientation: res.orientation
+                    };
+                }
+                if (Object.keys(res.missing).length === 0) {
+                    this.data.home = goal.name;
+                }
+                for (let block_name in res.missing) {
+                    this.temp_goals.push({
+                        name: block_name,
+                        quantity: res.missing[block_name]
+                    })
+                }
+                if (res.acted) {
+                    acted = true;
+                    this.last_goals[goal.name] = Object.keys(res.missing).length === 0;
+                    break;
+                }
+            }
+        }
+
+        if (!acted && this.data.do_set_goal)
+            await this.setGoal();
+    }
+
+    currentBuilding() {
+        let bot_pos = this.agent.bot.entity.position;
+        for (let name in this.data.built) {
+            let pos = this.data.built[name].position;
+            let offset = this.constructions[name].offset;
+            let sizex = this.constructions[name].blocks[0][0].length;
+            let sizez = this.constructions[name].blocks[0].length;
+            let sizey = this.constructions[name].blocks.length;
+            if (this.data.built[name].orientation % 2 === 1) [sizex, sizez] = [sizez, sizex];
+            if (bot_pos.x >= pos.x && bot_pos.x < pos.x + sizex &&
+                bot_pos.y >= pos.y + offset && bot_pos.y < pos.y + sizey + offset &&
+                bot_pos.z >= pos.z && bot_pos.z < pos.z + sizez) {
+                return name;
+            }
+        }
+        return null;
+    }
+
+    getBuildingDoor(name) {
+        if (name === null || this.data.built[name] === undefined) return null;
+        let door_x = null;
+        let door_z = null;
+        let door_y = null;
+        for (let y = 0; y < this.constructions[name].blocks.length; y++) {
+            for (let z = 0; z < this.constructions[name].blocks[y].length; z++) {
+                for (let x = 0; x < this.constructions[name].blocks[y][z].length; x++) {
+                    if (this.constructions[name].blocks[y][z][x] !== null &&
+                        this.constructions[name].blocks[y][z][x].includes('door')) {
+                        door_x = x;
+                        door_z = z;
+                        door_y = y;
+                        break;
+                    }
+                }
+                if (door_x !== null) break;
+            }
+            if (door_x !== null) break;
+        }
+        if (door_x === null) return null;
+
+        let sizex = this.constructions[name].blocks[0][0].length;
+        let sizez = this.constructions[name].blocks[0].length;
+        let orientation = 4 - this.data.built[name].orientation; // this conversion is opposite
+        if (orientation == 4) orientation = 0;
+        [door_x, door_z] = rotateXZ(door_x, door_z, orientation, sizex, sizez);
+        door_y += this.constructions[name].offset;
+
+        return {
+            x: this.data.built[name].position.x + door_x,
+            y: this.data.built[name].position.y + door_y,
+            z: this.data.built[name].position.z + door_z
+        };
+    }
+}

src/agent/npc/data.js

@@ -0,0 +1,50 @@
+export class NPCData {
+    constructor() {
+        this.goals = [];
+        this.curr_goal = null;
+        this.built = {};
+        this.home = null;
+        this.do_routine = false;
+        this.do_set_goal = false;
+    }
+
+    toObject() {
+        let obj = {};
+        if (this.goals.length > 0)
+            obj.goals = this.goals;
+        if (this.curr_goal)
+            obj.curr_goal = this.curr_goal;
+        if (Object.keys(this.built).length > 0)
+            obj.built = this.built;
+        if (this.home)
+            obj.home = this.home;
+        obj.do_routine = this.do_routine;
+        obj.do_set_goal = this.do_set_goal;
+        return obj;
+    }
+
+    static fromObject(obj) {
+        let npc = new NPCData();
+        if (!obj) return npc;
+        if (obj.goals) {
+            npc.goals = [];
+            for (let goal of obj.goals) {
+                if (typeof goal === 'string')
+                    npc.goals.push({name: goal, quantity: 1});
+                else
+                    npc.goals.push({name: goal.name, quantity: goal.quantity});
+            }
+        }
+        if (obj.curr_goal)
+            npc.curr_goal = obj.curr_goal;
+        if (obj.built)
+            npc.built = obj.built;
+        if (obj.home)
+            npc.home = obj.home;
+        if (obj.do_routine !== undefined)
+            npc.do_routine = obj.do_routine;
+        if (obj.do_set_goal !== undefined)
+            npc.do_set_goal = obj.do_set_goal;
+        return npc;
+    }
+}

src/agent/npc/item_goal.js

@@ -0,0 +1,355 @@
+import * as skills from '../library/skills.js';
+import * as world from '../library/world.js';
+import * as mc from '../../utils/mcdata.js';
+import { itemSatisfied } from './utils.js';
+
+
+const blacklist = [
+    'coal_block',
+    'iron_block',
+    'gold_block',
+    'diamond_block',
+    'deepslate',
+    'blackstone',
+    'netherite',
+    '_wood',
+    'stripped_',
+    'crimson',
+    'warped',
+    'dye'
+]
+
+
+class ItemNode {
+    constructor(manager, wrapper, name) {
+        this.manager = manager;
+        this.wrapper = wrapper;
+        this.name = name;
+        this.type = '';
+        this.source = null;
+        this.prereq = null;
+        this.recipe = [];
+        this.fails = 0;
+    }
+
+    setRecipe(recipe) {
+        this.type = 'craft';
+        let size = 0;
+        this.recipe = [];
+        for (let [key, value] of Object.entries(recipe)) {
+            if (this.manager.nodes[key] === undefined)
+                this.manager.nodes[key] = new ItemWrapper(this.manager, this.wrapper, key);
+            this.recipe.push({node: this.manager.nodes[key], quantity: value});
+            size += value;
+        }
+        if (size > 4) {
+            if (this.manager.nodes['crafting_table'] === undefined)
+                this.manager.nodes['crafting_table'] = new ItemWrapper(this.manager, this.wrapper, 'crafting_table');
+            this.prereq = this.manager.nodes['crafting_table'];
+        }
+        return this;
+    }
+
+    setCollectable(source=null, tool=null) {
+        this.type = 'block';
+        if (source)
+            this.source = source;
+        else
+            this.source = this.name;
+        if (tool) {
+            if (this.manager.nodes[tool] === undefined)
+                this.manager.nodes[tool] = new ItemWrapper(this.manager, this.wrapper, tool);
+            this.prereq = this.manager.nodes[tool];
+        }
+        return this;
+    }
+
+    setSmeltable(source_item) {
+        this.type = 'smelt';
+        if (this.manager.nodes['furnace'] === undefined)
+            this.manager.nodes['furnace'] = new ItemWrapper(this.manager, this.wrapper, 'furnace');
+        this.prereq = this.manager.nodes['furnace'];
+
+        if (this.manager.nodes[source_item] === undefined)
+            this.manager.nodes[source_item] = new ItemWrapper(this.manager, this.wrapper, source_item);
+        if (this.manager.nodes['coal'] === undefined)
+            this.manager.nodes['coal'] = new ItemWrapper(this.manager, this.wrapper, 'coal');
+        this.recipe = [
+            {node: this.manager.nodes[source_item], quantity: 1},
+            {node: this.manager.nodes['coal'], quantity: 1}
+        ];
+        return this;
+    }
+
+    setHuntable(animal_source) {
+        this.type = 'hunt';
+        this.source = animal_source;
+        return this;
+    }
+
+    getChildren() {
+        let children = [...this.recipe];
+        if (this.prereq) {
+            children.push({node: this.prereq, quantity: 1});
+        }
+        return children;
+    }
+
+    isReady() {
+        for (let child of this.getChildren()) {
+            if (!child.node.isDone(child.quantity)) {
+                return false;
+            }
+        }
+        return true;
+    }
+
+    isDone(quantity=1) {
+        if (this.manager.goal.name === this.name)
+            return false;
+        return itemSatisfied(this.manager.agent.bot, this.name, quantity);
+    }
+
+    getDepth(q=1) {
+        if (this.isDone(q)) {
+            return 0;
+        }
+        let depth = 0;
+        for (let child of this.getChildren()) {
+            depth = Math.max(depth, child.node.getDepth(child.quantity));
+        }
+        return depth + 1;
+    }
+
+    getFails(q=1) {
+        if (this.isDone(q)) {
+            return 0;
+        }
+        let fails = 0;
+        for (let child of this.getChildren()) {
+            fails += child.node.getFails(child.quantity);
+        }
+        return fails + this.fails;
+    }
+
+    getNext(q=1) {
+        if (this.isDone(q))
+            return null;
+        if (this.isReady())
+            return {node: this, quantity: q};
+        for (let child of this.getChildren()) {
+            let res = child.node.getNext(child.quantity);
+            if (res)
+                return res;
+        }
+        return null;
+    }
+
+    async execute(quantity=1) {
+        if (!this.isReady()) {
+            this.fails += 1;
+            return;
+        }
+        let inventory = world.getInventoryCounts(this.manager.agent.bot);
+        let init_quantity = inventory[this.name] || 0;
+        if (this.type === 'block') {
+            await skills.collectBlock(this.manager.agent.bot, this.source, quantity, this.manager.agent.npc.getBuiltPositions());
+        } else if (this.type === 'smelt') {
+            let to_smelt_name = this.recipe[0].node.name;
+            let to_smelt_quantity = Math.min(quantity, inventory[to_smelt_name] || 1);
+            await skills.smeltItem(this.manager.agent.bot, to_smelt_name, to_smelt_quantity);
+        } else if (this.type === 'hunt') {
+            for (let i=0; i<quantity; i++) {
+                res = await skills.attackNearest(this.manager.agent.bot, this.source);
+                if (!res || this.manager.agent.bot.interrupt_code)
+                    break;
+            }
+        } else if (this.type === 'craft') {
+            await skills.craftRecipe(this.manager.agent.bot, this.name, quantity);
+        }
+        let final_quantity = world.getInventoryCounts(this.manager.agent.bot)[this.name] || 0;
+        if (final_quantity <= init_quantity) {
+            this.fails += 1;
+        }
+    }
+}
+
+
+class ItemWrapper {
+    constructor(manager, parent, name) {
+        this.manager = manager;
+        this.name = name;
+        this.parent = parent;
+        this.methods = [];
+
+        let blacklisted = false;
+        for (let match of blacklist) {
+            if (name.includes(match)) {
+                blacklisted = true;
+                break;
+            }
+        }
+
+        if (!blacklisted && !this.containsCircularDependency()) {
+            this.createChildren();
+        }
+    }
+
+    add_method(method) {
+        for (let child of method.getChildren()) {
+            if (child.node.methods.length === 0)
+                return;
+        }
+        this.methods.push(method);
+    }
+
+    createChildren() {
+        let recipes = mc.getItemCraftingRecipes(this.name).map(([recipe, craftedCount]) => recipe);
+        if (recipes) {
+            for (let recipe of recipes) {
+                let includes_blacklisted = false;
+                for (let ingredient in recipe) {
+                    for (let match of blacklist) {
+                        if (ingredient.includes(match)) {
+                            includes_blacklisted = true;
+                            break;
+                        }
+                    }
+                    if (includes_blacklisted) break;
+                }
+                if (includes_blacklisted) continue;
+                this.add_method(new ItemNode(this.manager, this, this.name).setRecipe(recipe))
+            }
+        }
+
+        let block_sources = mc.getItemBlockSources(this.name);
+        if (block_sources.length > 0 && this.name !== 'torch' && !this.name.includes('bed')) {  // Do not collect placed torches or beds
+            for (let block_source of block_sources) {
+                if (block_source === 'grass_block') continue;  // Dirt nodes will collect grass blocks
+                let tool = mc.getBlockTool(block_source);
+                this.add_method(new ItemNode(this.manager, this, this.name).setCollectable(block_source, tool));
+            }
+        }
+
+        let smeltingIngredient = mc.getItemSmeltingIngredient(this.name);
+        if (smeltingIngredient) {
+            this.add_method(new ItemNode(this.manager, this, this.name).setSmeltable(smeltingIngredient));
+        }
+
+        let animal_source = mc.getItemAnimalSource(this.name);
+        if (animal_source) {
+            this.add_method(new ItemNode(this.manager, this, this.name).setHuntable(animal_source));
+        }
+    }
+
+    containsCircularDependency() {
+        let p = this.parent;
+        while (p) {
+            if (p.name === this.name) {
+                return true;
+            }
+            p = p.parent;
+        }
+        return false;
+    }
+
+    getBestMethod(q=1) {
+        let best_cost = -1;
+        let best_method = null;
+        for (let method of this.methods) {
+            let cost = method.getDepth(q) + method.getFails(q);
+            if (best_cost == -1 || cost < best_cost) {
+                best_cost = cost;
+                best_method = method;
+            }
+        }
+        return best_method
+    }
+
+    isDone(q=1) {
+        if (this.methods.length === 0)
+            return false;
+        return this.getBestMethod(q).isDone(q);
+    }
+
+    getDepth(q=1) {
+        if (this.methods.length === 0)
+            return 0;
+        return this.getBestMethod(q).getDepth(q);
+    }
+
+    getFails(q=1) {
+        if (this.methods.length === 0)
+            return 0;
+        return this.getBestMethod(q).getFails(q);
+    }
+
+    getNext(q=1) {
+        if (this.methods.length === 0)
+            return null;
+        return this.getBestMethod(q).getNext(q);
+    }
+}
+
+
+export class ItemGoal {
+    constructor(agent) {
+        this.agent = agent;
+        this.goal = null;
+        this.nodes = {};
+        this.failed = [];
+    }
+
+    async executeNext(item_name, item_quantity=1) {
+        if (this.nodes[item_name] === undefined)
+            this.nodes[item_name] = new ItemWrapper(this, null, item_name);
+        this.goal = this.nodes[item_name];
+
+        // Get next goal to execute
+        let next_info = this.goal.getNext(item_quantity);
+        if (!next_info) {
+            console.log(`Invalid item goal ${this.goal.name}`);
+            return false;
+        }
+        let next = next_info.node;
+        let quantity = next_info.quantity;
+
+        // Prevent unnecessary attempts to obtain blocks that are not nearby
+        if (next.type === 'block' && !world.getNearbyBlockTypes(this.agent.bot).includes(next.source) ||
+                next.type === 'hunt' && !world.getNearbyEntityTypes(this.agent.bot).includes(next.source)) {
+            next.fails += 1;
+
+            // If the bot has failed to obtain the block before, explore
+            if (this.failed.includes(next.name)) {
+                this.failed = this.failed.filter((item) => item !== next.name);
+                await this.agent.actions.runAction('itemGoal:explore', async () => {
+                    await skills.moveAway(this.agent.bot, 8);
+                });
+            } else {
+                this.failed.push(next.name);
+                await new Promise((resolve) => setTimeout(resolve, 500));
+                this.agent.bot.emit('idle');
+            }
+            return false;
+        }
+
+        // Wait for the bot to be idle before attempting to execute the next goal
+        if (!this.agent.isIdle())
+            return false;
+
+        // Execute the next goal
+        let init_quantity = world.getInventoryCounts(this.agent.bot)[next.name] || 0;
+        await this.agent.actions.runAction('itemGoal:next', async () => {
+            await next.execute(quantity);
+        });
+        let final_quantity = world.getInventoryCounts(this.agent.bot)[next.name] || 0;
+
+        // Log the result of the goal attempt
+        if (final_quantity > init_quantity) {
+            console.log(`Successfully obtained ${next.name} for goal ${this.goal.name}`);
+        } else {
+            console.log(`Failed to obtain ${next.name} for goal ${this.goal.name}`);
+        }
+        return final_quantity > init_quantity;
+    }
+}

src/agent/npc/utils.js

@@ -0,0 +1,126 @@
+import * as world from '../library/world.js';
+import * as mc from '../../utils/mcdata.js';
+
+
+export function getTypeOfGeneric(bot, block_name) {
+    // Get type of wooden block
+    if (mc.MATCHING_WOOD_BLOCKS.includes(block_name)) {
+
+        // Return most common wood type in inventory
+        let type_count = {};
+        let max_count = 0;
+        let max_type = null;
+        let inventory = world.getInventoryCounts(bot);
+        for (const item in inventory) {
+            for (const wood of mc.WOOD_TYPES) {
+                if (item.includes(wood)) {
+                    if (type_count[wood] === undefined)
+                        type_count[wood] = 0;
+                    type_count[wood] += inventory[item];
+                    if (type_count[wood] > max_count) {
+                        max_count = type_count[wood];
+                        max_type = wood;
+                    }
+                }
+            }
+        }
+        if (max_type !== null)
+            return max_type + '_' + block_name;
+
+        // Return nearest wood type
+        let log_types = mc.WOOD_TYPES.map((wood) => wood + '_log');
+        let blocks = world.getNearestBlocks(bot, log_types, 16, 1);
+        if (blocks.length > 0) {
+            let wood = blocks[0].name.split('_')[0];
+            return wood + '_' + block_name;
+        }
+
+        // Return oak
+        return 'oak_' + block_name;
+    }
+
+    // Get type of bed
+    if (block_name === 'bed') {
+
+        // Return most common wool type in inventory
+        let type_count = {};
+        let max_count = 0;
+        let max_type = null;
+        let inventory = world.getInventoryCounts(bot);
+        for (const item in inventory) {
+            for (const color of mc.WOOL_COLORS) {
+                if (item === color + '_wool') {
+                    if (type_count[color] === undefined)
+                        type_count[color] = 0;
+                    type_count[color] += inventory[item];
+                    if (type_count[color] > max_count) {
+                        max_count = type_count[color];
+                        max_type = color;
+                    }
+                }
+            }
+        }
+        if (max_type !== null)
+            return max_type + '_' + block_name;
+
+        // Return white
+        return 'white_' + block_name;
+    }
+    return block_name;
+}
+
+
+export function blockSatisfied(target_name, block) {
+    if (target_name == 'dirt') {
+        return block.name == 'dirt' || block.name == 'grass_block';
+    } else if (mc.MATCHING_WOOD_BLOCKS.includes(target_name)) {
+        return block.name.endsWith(target_name);
+    } else if (target_name == 'bed') {
+        return block.name.endsWith('bed');
+    } else if (target_name == 'torch') {
+        return block.name.includes('torch');
+    }
+    return block.name == target_name;
+}
+
+
+export function itemSatisfied(bot, item, quantity=1) {
+    let qualifying = [item];
+    if (item.includes('pickaxe') || 
+            item.includes('axe') || 
+            item.includes('shovel') ||
+            item.includes('hoe') ||
+            item.includes('sword')) {
+        let material = item.split('_')[0];
+        let type = item.split('_')[1];
+        if (material === 'wooden') {
+            qualifying.push('stone_' + type);
+            qualifying.push('iron_' + type);
+            qualifying.push('gold_' + type);
+            qualifying.push('diamond_' + type);
+        } else if (material === 'stone') {
+            qualifying.push('iron_' + type);
+            qualifying.push('gold_' + type);
+            qualifying.push('diamond_' + type);
+        } else if (material === 'iron') {
+            qualifying.push('gold_' + type);
+            qualifying.push('diamond_' + type);
+        } else if (material === 'gold') {
+            qualifying.push('diamond_' + type);
+        }
+    }
+    for (let item of qualifying) {
+        if (world.getInventoryCounts(bot)[item] >= quantity) {
+            return true;
+        }
+    }
+    return false;
+}
+
+
+export function rotateXZ(x, z, orientation, sizex, sizez) {
+    if (orientation === 0) return [x, z];
+    if (orientation === 1) return [z, sizex-x-1];
+    if (orientation === 2) return [sizex-x-1, sizez-z-1];
+    if (orientation === 3) return [sizez-z-1, x];
+}

src/agent/self_prompter.js

@@ -0,0 +1,114 @@
+export class SelfPrompter {
+    constructor(agent) {
+        this.agent = agent;
+        this.on = false;
+        this.loop_active = false;
+        this.interrupt = false;
+        this.prompt = '';
+        this.idle_time = 0;
+        this.cooldown = 2000;
+    }
+
+    start(prompt) {
+        console.log('Self-prompting started.');
+        if (!prompt) {
+            if (!this.prompt)
+                return 'No prompt specified. Ignoring request.';
+            prompt = this.prompt;
+        }
+        if (this.on) {
+            this.prompt = prompt;
+        }
+        this.on = true;
+        this.prompt = prompt;
+        this.startLoop();
+    }
+
+    setPrompt(prompt) {
+        this.prompt = prompt;
+    }
+
+    async startLoop() {
+        if (this.loop_active) {
+            console.warn('Self-prompt loop is already active. Ignoring request.');
+            return;
+        }
+        console.log('starting self-prompt loop')
+        this.loop_active = true;
+        let no_command_count = 0;
+        const MAX_NO_COMMAND = 3;
+        while (!this.interrupt) {
+            const msg = `You are self-prompting with the goal: '${this.prompt}'. Your next response MUST contain a command with this syntax: !commandName. Respond:`;
+            
+            let used_command = await this.agent.handleMessage('system', msg, -1);
+            if (!used_command) {
+                no_command_count++;
+                if (no_command_count >= MAX_NO_COMMAND) {
+                    let out = `Agent did not use command in the last ${MAX_NO_COMMAND} auto-prompts. Stopping auto-prompting.`;
+                    this.agent.openChat(out);
+                    console.warn(out);
+                    this.on = false;
+                    break;
+                }
+            }
+            else {
+                no_command_count = 0;
+                await new Promise(r => setTimeout(r, this.cooldown));
+            }
+        }
+        console.log('self prompt loop stopped')
+        this.loop_active = false;
+        this.interrupt = false;
+    }
+
+    update(delta) {
+        // automatically restarts loop
+        if (this.on && !this.loop_active && !this.interrupt) {
+            if (this.agent.isIdle())
+                this.idle_time += delta;
+            else
+                this.idle_time = 0;
+
+            if (this.idle_time >= this.cooldown) {
+                console.log('Restarting self-prompting...');
+                this.startLoop();
+                this.idle_time = 0;
+            }
+        }
+        else {
+            this.idle_time = 0;
+        }
+    }
+
+    async stopLoop() {
+        // you can call this without await if you don't need to wait for it to finish
+        if (this.interrupt)
+            return;
+        console.log('stopping self-prompt loop')
+        this.interrupt = true;
+        while (this.loop_active) {
+            await new Promise(r => setTimeout(r, 500));
+        }
+        this.interrupt = false;
+    }
+
+    async stop(stop_action=true) {
+        this.interrupt = true;
+        if (stop_action)
+            await this.agent.actions.stop();
+        await this.stopLoop();
+        this.on = false;
+    }
+
+    shouldInterrupt(is_self_prompt) { // to be called from handleMessage
+        return is_self_prompt && this.on && this.interrupt;
+    }
+
+    handleUserPromptedCmd(is_self_prompt, is_action) {
+        // if a user messages and the bot responds with an action, stop the self-prompt loop
+        if (!is_self_prompt && is_action) {
+            this.stopLoop();
+            // this stops it from responding from the handlemessage loop and the self-prompt loop at the same time
+        }
+    }
+}

src/agent/tasks.js

@@ -0,0 +1,196 @@
+import { readFileSync } from 'fs';
+import { executeCommand } from './commands/index.js';
+import { getPosition } from './library/world.js'
+import settings from '../../settings.js';
+
+
+export class TaskValidator {
+    constructor(data, agent) {
+        this.target = data.target;
+        this.number_of_target = data.number_of_target;
+        this.agent = agent;
+    }
+
+    validate() {
+        try{
+            let valid = false;
+            let total_targets = 0;
+            this.agent.bot.inventory.slots.forEach((slot) => {
+                if (slot && slot.name.toLowerCase() === this.target) {
+                    total_targets += slot.count;
+                }
+                if (slot && slot.name.toLowerCase() === this.target && slot.count >= this.number_of_target) {
+                    valid = true;
+                    console.log('Task is complete');
+                }
+            });
+            if (total_targets >= this.number_of_target) {
+                valid = true;
+                console.log('Task is complete');
+            }
+            return valid;
+        } catch (error) {
+            console.error('Error validating task:', error);
+            return false;
+        }
+    }
+}
+
+
+export class Task {
+    constructor(agent, task_path, task_id) {
+        this.agent = agent;
+        this.data = null;
+        this.taskTimeout = 300;
+        this.taskStartTime = Date.now();
+        this.validator = null;
+        this.blocked_actions = [];
+        if (task_path && task_id) {
+            this.data = this.loadTask(task_path, task_id);
+            this.taskTimeout = this.data.timeout || 300;
+            this.taskStartTime = Date.now();
+            this.validator = new TaskValidator(this.data, this.agent);
+            this.blocked_actions = this.data.blocked_actions || [];
+            this.restrict_to_inventory = !!this.data.restrict_to_inventory;
+            if (this.data.goal)
+                this.blocked_actions.push('!endGoal');
+            if (this.data.conversation)
+                this.blocked_actions.push('!endConversation');
+        }
+    }
+
+    loadTask(task_path, task_id) {
+        try {
+            const tasksFile = readFileSync(task_path, 'utf8');
+            const tasks = JSON.parse(tasksFile);
+            const task = tasks[task_id];
+            if (!task) {
+                throw new Error(`Task ${task_id} not found`);
+            }
+            if ((!task.agent_count || task.agent_count <= 1) && this.agent.count_id > 0) {
+                task = null;
+            }
+
+            return task;
+        } catch (error) {
+            console.error('Error loading task:', error);
+            process.exit(1);
+        }
+    }
+
+    isDone() {
+        if (this.validator && this.validator.validate())
+            return {"message": 'Task successful', "code": 2};
+        // TODO check for other terminal conditions
+        // if (this.task.goal && !this.self_prompter.on)
+        //     return {"message": 'Agent ended goal', "code": 3};
+        // if (this.task.conversation && !inConversation())
+        //     return {"message": 'Agent ended conversation', "code": 3};
+        if (this.taskTimeout) {
+            const elapsedTime = (Date.now() - this.taskStartTime) / 1000;
+            if (elapsedTime >= this.taskTimeout) {
+                console.log('Task timeout reached. Task unsuccessful.');
+                return {"message": 'Task timeout reached', "code": 4};
+            }
+        }
+        return false;
+    }
+
+    async initBotTask() {
+        if (this.data === null)
+            return;
+        let bot = this.agent.bot;
+        let name = this.agent.name;
+    
+        bot.chat(`/clear ${name}`);
+        console.log(`Cleared ${name}'s inventory.`);
+        
+        //wait for a bit so inventory is cleared
+        await new Promise((resolve) => setTimeout(resolve, 500));
+    
+        if (this.data.agent_count > 1) {
+            let initial_inventory = this.data.initial_inventory[this.agent.count_id.toString()];
+            console.log("Initial inventory:", initial_inventory);
+        } else if (this.data) {
+            console.log("Initial inventory:", this.data.initial_inventory);
+            let initial_inventory = this.data.initial_inventory;
+        }
+    
+        if ("initial_inventory" in this.data) {
+            console.log("Setting inventory...");
+            console.log("Inventory to set:", initial_inventory);
+            for (let key of Object.keys(initial_inventory)) {
+                console.log('Giving item:', key);
+                bot.chat(`/give ${name} ${key} ${initial_inventory[key]}`);
+            };
+            //wait for a bit so inventory is set
+            await new Promise((resolve) => setTimeout(resolve, 500));
+            console.log("Done giving inventory items.");
+        }
+        // Function to generate random numbers
+    
+        function getRandomOffset(range) {
+            return Math.floor(Math.random() * (range * 2 + 1)) - range;
+        }
+    
+        let human_player_name = null;
+        let available_agents = settings.profiles.map((p) => JSON.parse(readFileSync(p, 'utf8')).name);  // TODO this does not work with command line args
+    
+        // Finding if there is a human player on the server
+        for (const playerName in bot.players) {
+            const player = bot.players[playerName];
+            if (!available_agents.some((n) => n === playerName)) {
+                console.log('Found human player:', player.username);
+                human_player_name = player.username
+                break;
+            }
+        }
+
+        // If there are multiple human players, teleport to the first one
+    
+        // teleport near a human player if found by default
+    
+        if (human_player_name) {
+            console.log(`Teleporting ${name} to human ${human_player_name}`)
+            bot.chat(`/tp ${name} ${human_player_name}`) // teleport on top of the human player
+    
+        }
+        await new Promise((resolve) => setTimeout(resolve, 200));
+    
+        // now all bots are teleport on top of each other (which kinda looks ugly)
+        // Thus, we need to teleport them to random distances to make it look better
+    
+        /*
+        Note : We don't want randomness for construction task as the reference point matters a lot.
+        Another reason for no randomness for construction task is because, often times the user would fly in the air,
+        then set a random block to dirt and teleport the bot to stand on that block for starting the construction,
+        This was done by MaxRobinson in one of the youtube videos.
+        */
+    
+        if (this.data.type !== 'construction') {
+            const pos = getPosition(bot);
+            const xOffset = getRandomOffset(5);
+            const zOffset = getRandomOffset(5);
+            bot.chat(`/tp ${name} ${Math.floor(pos.x + xOffset)} ${pos.y + 3} ${Math.floor(pos.z + zOffset)}`);
+            await new Promise((resolve) => setTimeout(resolve, 200));
+        }
+
+        if (this.data.agent_count && this.data.agent_count > 1) {
+            // TODO wait for other bots to join
+            await new Promise((resolve) => setTimeout(resolve, 10000));
+            if (available_agents.length < this.data.agent_count) {
+                console.log(`Missing ${this.data.agent_count - available_agents.length} bot(s).`);
+                this.agent.killAll();
+            }
+        }
+
+        if (this.data.goal) {
+            await executeCommand(this.agent, `!goal("${this.data.goal}")`);
+        }
+    
+        if (this.data.conversation && this.agent.count_id === 0) {
+            let other_name = available_agents.filter(n => n !== name)[0];
+            await executeCommand(this.agent, `!startConversation("${other_name}", "${this.data.conversation}")`);
+        }
+    }    
+}

src/agent/viewer.js

@@ -0,0 +1,8 @@
+import settings from '../../settings.js';
+import prismarineViewer from 'prismarine-viewer';
+const mineflayerViewer = prismarineViewer.mineflayer;
+
+export function addViewer(bot, count_id) {
+    if (settings.show_bot_views)
+        mineflayerViewer(bot, { port: 3000+count_id, firstPerson: true, });
+}

src/models/claude.js

@@ -0,0 +1,50 @@
+import Anthropic from '@anthropic-ai/sdk';
+import { strictFormat } from '../utils/text.js';
+import { getKey } from '../utils/keys.js';
+
+export class Claude {
+    constructor(model_name, url, params) {
+        this.model_name = model_name;
+        this.params = params || {};
+
+        let config = {};
+        if (url)
+            config.baseURL = url;
+        
+        config.apiKey = getKey('ANTHROPIC_API_KEY');
+
+        this.anthropic = new Anthropic(config);
+    }
+
+    async sendRequest(turns, systemMessage) {
+        const messages = strictFormat(turns);
+        let res = null;
+        try {
+            console.log('Awaiting anthropic api response...')
+            if (!this.params.max_tokens) {
+                this.params.max_tokens = 4096;
+            }
+            const resp = await this.anthropic.messages.create({
+                model: this.model_name || "claude-3-sonnet-20240229",
+                system: systemMessage,
+                messages: messages,
+                ...(this.params || {})
+            });
+
+            console.log('Received.')
+            res = resp.content[0].text;
+        }
+        catch (err) {
+            console.log(err);
+            res = 'My brain disconnected, try again.';
+        }
+        return res;
+    }
+
+    async embed(text) {
+        throw new Error('Embeddings are not supported by Claude.');
+    }
+}
+
+
+

src/models/deepseek.js

@@ -0,0 +1,58 @@
+import OpenAIApi from 'openai';
+import { getKey, hasKey } from '../utils/keys.js';
+import { strictFormat } from '../utils/text.js';
+
+export class DeepSeek {
+    constructor(model_name, url, params) {
+        this.model_name = model_name;
+        this.params = params;
+
+        let config = {};
+
+        config.baseURL = url || 'https://api.deepseek.com';
+        config.apiKey = getKey('DEEPSEEK_API_KEY');
+
+        this.openai = new OpenAIApi(config);
+    }
+
+    async sendRequest(turns, systemMessage, stop_seq='***') {
+        let messages = [{'role': 'system', 'content': systemMessage}].concat(turns);
+
+        messages = strictFormat(messages);
+
+        const pack = {
+            model: this.model_name || "deepseek-chat",
+            messages,
+            stop: stop_seq,
+            ...(this.params || {})
+        };
+
+        let res = null;
+        try {
+            console.log('Awaiting deepseek api response...')
+            // console.log('Messages:', messages);
+            let completion = await this.openai.chat.completions.create(pack);
+            if (completion.choices[0].finish_reason == 'length')
+                throw new Error('Context length exceeded'); 
+            console.log('Received.')
+            res = completion.choices[0].message.content;
+        }
+        catch (err) {
+            if ((err.message == 'Context length exceeded' || err.code == 'context_length_exceeded') && turns.length > 1) {
+                console.log('Context length exceeded, trying again with shorter context.');
+                return await this.sendRequest(turns.slice(1), systemMessage, stop_seq);
+            } else {
+                console.log(err);
+                res = 'My brain disconnected, try again.';
+            }
+        }
+        return res;
+    }
+
+    async embed(text) {
+        throw new Error('Embeddings are not supported by Deepseek.');
+    }
+}
+
+
+

src/models/gemini.js

@@ -0,0 +1,116 @@
+import { GoogleGenerativeAI } from '@google/generative-ai';
+import { toSinglePrompt, strictFormat } from '../utils/text.js';
+import { getKey } from '../utils/keys.js';
+
+export class Gemini {
+    constructor(model_name, url, params) {
+        this.model_name = model_name;
+        this.params = params;
+        this.url = url;
+        this.safetySettings = [
+            {
+                "category": "HARM_CATEGORY_DANGEROUS",
+                "threshold": "BLOCK_NONE",
+            },
+            {
+                "category": "HARM_CATEGORY_HARASSMENT",
+                "threshold": "BLOCK_NONE",
+            },
+            {
+                "category": "HARM_CATEGORY_HATE_SPEECH",
+                "threshold": "BLOCK_NONE",
+            },
+            {
+                "category": "HARM_CATEGORY_SEXUALLY_EXPLICIT",
+                "threshold": "BLOCK_NONE",
+            },
+            {
+                "category": "HARM_CATEGORY_DANGEROUS_CONTENT",
+                "threshold": "BLOCK_NONE",
+            },
+        ];
+
+        this.genAI = new GoogleGenerativeAI(getKey('GEMINI_API_KEY'));
+    }
+
+    async sendRequest(turns, systemMessage) {
+        let model;
+        const modelConfig = {
+            model: this.model_name || "gemini-1.5-flash",
+            // systemInstruction does not work bc google is trash
+        };
+
+        if (this.url) {
+            model = this.genAI.getGenerativeModel(
+                modelConfig,
+                { baseUrl: this.url },
+                { safetySettings: this.safetySettings }
+            );
+        } else {
+            model = this.genAI.getGenerativeModel(
+                modelConfig,
+                { safetySettings: this.safetySettings }
+            );
+        }
+
+        console.log('Awaiting Google API response...');
+
+        // Prepend system message and format turns cause why not
+        turns.unshift({ role: 'system', content: systemMessage });
+        turns = strictFormat(turns);
+        let contents = [];
+        for (let turn of turns) {
+            contents.push({
+                role: turn.role === 'assistant' ? 'model' : 'user',
+                parts: [{ text: turn.content }]
+            });
+        }
+
+        const result = await model.generateContent({
+            contents,
+            generationConfig: {
+                ...(this.params || {})
+            }
+        });
+        const response = await result.response;
+        let text;
+
+        // Handle "thinking" models since they smart 
+        if (this.model_name && this.model_name.includes("thinking")) {
+            if (
+                response.candidates &&
+                response.candidates.length > 0 &&
+                response.candidates[0].content &&
+                response.candidates[0].content.parts &&
+                response.candidates[0].content.parts.length > 1
+            ) {
+                text = response.candidates[0].content.parts[1].text;
+            } else {
+                console.warn("Unexpected response structure for thinking model:", response);
+                text = response.text();
+            }
+        } else {
+            text = response.text();
+        }
+
+        console.log('Received.');
+        return text;
+    }
+
+    async embed(text) {
+        let model;
+        if (this.url) {
+            model = this.genAI.getGenerativeModel(
+                { model: "text-embedding-004" },
+                { baseUrl: this.url }
+            );
+        } else {
+            model = this.genAI.getGenerativeModel(
+                { model: "text-embedding-004" }
+            );
+        }
+
+        const result = await model.embedContent(text);
+        return result.embedding.values;
+    }
+}

src/models/glhf.js

@@ -0,0 +1,62 @@
+import OpenAIApi from 'openai';
+import { getKey } from '../utils/keys.js';
+
+// glhf doesn't supply an SDK for their models, but fully supports OpenAI SDKs
+export class glhf {
+    constructor(model_name, url) {
+        this.model_name = model_name;
+
+        // Retrieve the API key from keys.json
+        const apiKey = getKey('GHLF_API_KEY');
+        if (!apiKey) {
+            throw new Error('API key not found. Please check keys.json and ensure GHLF_API_KEY is defined.');
+        }
+
+        // Configure OpenAIApi with the retrieved API key and base URL
+        this.openai = new OpenAIApi({
+            apiKey,
+            baseURL: url || "https://glhf.chat/api/openai/v1"
+        });
+    }
+
+    async sendRequest(turns, systemMessage, stop_seq = '***') {
+        // Construct the message array for the API request
+        let messages = [{ 'role': 'system', 'content': systemMessage }].concat(turns);
+
+        const pack = {
+            model: this.model_name || "hf:meta-llama/Llama-3.1-405B-Instruct",
+            messages,
+            stop: [stop_seq]
+        };
+
+        let res = null;
+        try {
+            console.log('Awaiting glhf.chat API response...');
+            // Uncomment the line below if you need to debug the messages
+            // console.log('Messages:', messages);
+
+            let completion = await this.openai.chat.completions.create(pack);
+            if (completion.choices[0].finish_reason === 'length') {
+                throw new Error('Context length exceeded');
+            }
+
+            console.log('Received.');
+            res = completion.choices[0].message.content;
+        } catch (err) {
+            if ((err.message === 'Context length exceeded' || err.code === 'context_length_exceeded') && turns.length > 1) {
+                console.log('Context length exceeded, trying again with shorter context.');
+                return await this.sendRequest(turns.slice(1), systemMessage, stop_seq);
+            } else {
+                console.log(err);
+                res = 'My brain disconnected, try again.';
+            }
+        }
+
+        // Replace special tokens in the response
+        return res.replace(/<\|separator\|>/g, '*no response*');
+    }
+
+    async embed(text) {
+        throw new Error('Embeddings are not supported by glhf.');
+    }
+}

src/models/gpt.js

@@ -0,0 +1,72 @@
+import OpenAIApi from 'openai';
+import { getKey, hasKey } from '../utils/keys.js';
+import { strictFormat } from '../utils/text.js';
+
+export class GPT {
+    constructor(model_name, url, params) {
+        this.model_name = model_name;
+        this.params = params;
+
+        let config = {};
+        if (url)
+            config.baseURL = url;
+
+        if (hasKey('OPENAI_ORG_ID'))
+            config.organization = getKey('OPENAI_ORG_ID');
+
+        config.apiKey = getKey('OPENAI_API_KEY');
+
+        this.openai = new OpenAIApi(config);
+    }
+
+    async sendRequest(turns, systemMessage, stop_seq='***') {
+        let messages = [{'role': 'system', 'content': systemMessage}].concat(turns);
+
+        const pack = {
+            model: this.model_name || "gpt-3.5-turbo",
+            messages,
+            stop: stop_seq,
+            ...(this.params || {})
+        };
+        if (this.model_name.includes('o1')) {
+            pack.messages = strictFormat(messages);
+            delete pack.stop;
+        }
+
+        let res = null;
+
+        try {
+            console.log('Awaiting openai api response from model', this.model_name)
+            // console.log('Messages:', messages);
+            let completion = await this.openai.chat.completions.create(pack);
+            if (completion.choices[0].finish_reason == 'length')
+                throw new Error('Context length exceeded'); 
+            console.log('Received.')
+            res = completion.choices[0].message.content;
+        }
+        catch (err) {
+            if ((err.message == 'Context length exceeded' || err.code == 'context_length_exceeded') && turns.length > 1) {
+                console.log('Context length exceeded, trying again with shorter context.');
+                return await this.sendRequest(turns.slice(1), systemMessage, stop_seq);
+            } else {
+                console.log(err);
+                res = 'My brain disconnected, try again.';
+            }
+        }
+        return res;
+    }
+
+    async embed(text) {
+        if (text.length > 8191)
+            text = text.slice(0, 8191);
+        const embedding = await this.openai.embeddings.create({
+            model: this.model_name || "text-embedding-3-small",
+            input: text,
+            encoding_format: "float",
+        });
+        return embedding.data[0].embedding;
+    }
+}
+
+
+

src/models/grok.js

@@ -0,0 +1,61 @@
+import OpenAIApi from 'openai';
+import { getKey } from '../utils/keys.js';
+
+// xAI doesn't supply a SDK for their models, but fully supports OpenAI and Anthropic SDKs
+export class Grok {
+    constructor(model_name, url, params) {
+        this.model_name = model_name;
+        this.url = url;
+        this.params = params;
+
+        let config = {};
+        if (url)
+            config.baseURL = url;
+        else
+            config.baseURL = "https://api.x.ai/v1"
+
+        config.apiKey = getKey('XAI_API_KEY');
+
+        this.openai = new OpenAIApi(config);
+    }
+
+    async sendRequest(turns, systemMessage, stop_seq='***') {
+        let messages = [{'role': 'system', 'content': systemMessage}].concat(turns);
+
+        const pack = {
+            model: this.model_name || "grok-beta",
+            messages,
+            stop: [stop_seq],
+            ...(this.params || {})
+        };
+
+        let res = null;
+        try {
+            console.log('Awaiting xai api response...')
+            ///console.log('Messages:', messages);
+            let completion = await this.openai.chat.completions.create(pack);
+            if (completion.choices[0].finish_reason == 'length')
+                throw new Error('Context length exceeded'); 
+            console.log('Received.')
+            res = completion.choices[0].message.content;
+        }
+        catch (err) {
+            if ((err.message == 'Context length exceeded' || err.code == 'context_length_exceeded') && turns.length > 1) {
+                console.log('Context length exceeded, trying again with shorter context.');
+                return await this.sendRequest(turns.slice(1), systemMessage, stop_seq);
+            } else {
+                console.log(err);
+                res = 'My brain disconnected, try again.';
+            }
+        }
+        // sometimes outputs special token <|separator|>, just replace it
+        return res.replace(/<\|separator\|>/g, '*no response*');
+    }
+    
+    async embed(text) {
+        throw new Error('Embeddings are not supported by Grok.');
+    }
+}
+
+
+

src/models/groq.js

@@ -0,0 +1,86 @@
+import Groq from 'groq-sdk'
+import { getKey } from '../utils/keys.js';
+
+// Umbrella class for Mixtral, LLama, Gemma...
+export class GroqCloudAPI {
+  constructor(model_name, url, params) {
+    this.model_name = model_name;
+    this.url = url;
+    this.params = params || {};
+    // Groq Cloud does not support custom URLs; warn if provided
+    if (this.url) {
+      console.warn("Groq Cloud has no implementation for custom URLs. Ignoring provided URL.");
+    }
+    this.groq = new Groq({ apiKey: getKey('GROQCLOUD_API_KEY') });
+  }
+
+  async sendRequest(turns, systemMessage, stop_seq = null) {
+    const maxAttempts = 5;
+    let attempt = 0;
+    let finalRes = null;
+    const messages = [{ role: "system", content: systemMessage }].concat(turns);
+
+    while (attempt < maxAttempts) {
+      attempt++;
+      let res = null;
+      try {
+        console.log(`Awaiting Groq response... (model: ${this.model_name || "mixtral-8x7b-32768"}, attempt: ${attempt})`);
+        if (!this.params.max_tokens) {
+          this.params.max_tokens = 16384;
+        }
+        // Create the streaming chat completion request
+        const completion = await this.groq.chat.completions.create({
+          messages: messages,
+          model: this.model_name || "mixtral-8x7b-32768",
+          stream: true,
+          stop: stop_seq,
+          ...(this.params || {})
+        });
+
+        let temp_res = "";
+        // Aggregate streamed chunks into a full response
+        for await (const chunk of completion) {
+          temp_res += chunk.choices[0]?.delta?.content || '';
+        }
+        res = temp_res;
+      } catch (err) {
+        console.log(err);
+        res = "My brain just kinda stopped working. Try again.";
+      }
+
+      // If the model name includes "deepseek-r1", handle the <think> tags
+      if (this.model_name && this.model_name.toLowerCase().includes("deepseek-r1")) {
+        const hasOpenTag = res.includes("<think>");
+        const hasCloseTag = res.includes("</think>");
+
+        // If a partial <think> block is detected, log a warning and retry
+        if (hasOpenTag && !hasCloseTag) {
+          console.warn("Partial <think> block detected. Re-generating Groq request...");
+          continue;
+        }
+
+        // If only the closing tag is present, prepend an opening tag
+        if (hasCloseTag && !hasOpenTag) {
+          res = '<think>' + res;
+        }
+        // Remove the complete <think> block (and any content inside) from the response
+        res = res.replace(/<think>[\s\S]*?<\/think>/g, '').trim();
+      }
+
+      finalRes = res;
+      break; // Exit the loop once a valid response is obtained
+    }
+
+    if (finalRes == null) {
+      console.warn("Could not obtain a valid <think> block or normal response after max attempts.");
+      finalRes = "Response incomplete, please try again.";
+    }
+    finalRes = finalRes.replace(/<\|separator\|>/g, '*no response*');
+
+    return finalRes;
+  }
+
+  async embed(text) {
+    console.log("There is no support for embeddings in Groq support. However, the following text was provided: " + text);
+  }
+}

src/models/huggingface.js

@@ -0,0 +1,87 @@
+import { toSinglePrompt } from '../utils/text.js';
+import { getKey } from '../utils/keys.js';
+import { HfInference } from "@huggingface/inference";
+
+export class HuggingFace {
+  constructor(model_name, url, params) {
+    // Remove 'huggingface/' prefix if present
+    this.model_name = model_name.replace('huggingface/', '');
+    this.url = url;
+    this.params = params;
+
+    if (this.url) {
+      console.warn("Hugging Face doesn't support custom urls!");
+    }
+
+    this.huggingface = new HfInference(getKey('HUGGINGFACE_API_KEY'));
+  }
+
+  async sendRequest(turns, systemMessage) {
+    const stop_seq = '***';
+    // Build a single prompt from the conversation turns
+    const prompt = toSinglePrompt(turns, null, stop_seq);
+    // Fallback model if none was provided
+    const model_name = this.model_name || 'meta-llama/Meta-Llama-3-8B';
+    // Combine system message with the prompt
+    const input = systemMessage + "\n" + prompt;
+
+    // We'll try up to 5 times in case of partial <think> blocks for DeepSeek-R1 models.
+    const maxAttempts = 5;
+    let attempt = 0;
+    let finalRes = null;
+
+    while (attempt < maxAttempts) {
+      attempt++;
+      console.log(`Awaiting Hugging Face API response... (model: ${model_name}, attempt: ${attempt})`);
+      let res = '';
+      try {
+        // Consume the streaming response chunk by chunk
+        for await (const chunk of this.huggingface.chatCompletionStream({
+          model: model_name,
+          messages: [{ role: "user", content: input }],
+          ...(this.params || {})
+        })) {
+          res += (chunk.choices[0]?.delta?.content || "");
+        }
+      } catch (err) {
+        console.log(err);
+        res = 'My brain disconnected, try again.';
+        // Break out immediately; we only retry when handling partial <think> tags.
+        break;
+      }
+
+      // If the model is DeepSeek-R1, check for mismatched <think> blocks.
+      if (this.model_name && this.model_name.toLowerCase().includes("deepseek-r1")) {
+        const hasOpenTag = res.includes("<think>");
+        const hasCloseTag = res.includes("</think>");
+
+        // If there's a partial mismatch, warn and retry the entire request.
+        if ((hasOpenTag && !hasCloseTag) || (!hasOpenTag && hasCloseTag)) {
+          console.warn("Partial <think> block detected. Re-generating...");
+          continue;
+        }
+
+        // If both tags are present, remove the <think> block entirely.
+        if (hasOpenTag && hasCloseTag) {
+          res = res.replace(/<think>[\s\S]*?<\/think>/g, '').trim();
+        }
+      }
+
+      finalRes = res;
+      break; // Exit loop if we got a valid response.
+    }
+
+    // If no valid response was obtained after max attempts, assign a fallback.
+    if (finalRes == null) {
+      console.warn("Could not get a valid <think> block or normal response after max attempts.");
+      finalRes = 'Response incomplete, please try again.';
+    }
+    console.log('Received.');
+    console.log(finalRes);
+    return finalRes;
+  }
+
+  async embed(text) {
+    throw new Error('Embeddings are not supported by HuggingFace.');
+  }
+}

src/models/hyperbolic.js

@@ -0,0 +1,92 @@
+import { getKey } from '../utils/keys.js';
+
+
+/*
+ * 
+ * Yes, this code was written by an Ai. It was written by GPT-o1 and tested :)
+ * 
+ */
+
+export class hyperbolic {
+    constructor(modelName, apiUrl) {
+        this.modelName = modelName || "deepseek-ai/DeepSeek-V3";
+        this.apiUrl = apiUrl || "https://api.hyperbolic.xyz/v1/chat/completions";
+
+        // Retrieve the Hyperbolic API key from keys.js
+        this.apiKey = getKey('HYPERBOLIC_API_KEY');
+        if (!this.apiKey) {
+            throw new Error('HYPERBOLIC_API_KEY not found. Check your keys.js file.');
+        }
+    }
+
+    /**
+     * Sends a chat completion request to the Hyperbolic endpoint.
+     *
+     * @param {Array} turns - An array of message objects, e.g. [{role: 'user', content: 'Hi'}].
+     * @param {string} systemMessage - The system prompt or instruction.
+     * @param {string} stopSeq - A string that represents a stopping sequence, default '***'.
+     * @returns {Promise<string>} - The content of the model's reply.
+     */
+    async sendRequest(turns, systemMessage, stopSeq = '***') {
+        // Prepare the messages with a system prompt at the beginning
+        const messages = [{ role: 'system', content: systemMessage }, ...turns];
+
+        // Build the request payload (mirroring your original structure)
+        const payload = {
+            model: this.modelName,
+            messages: messages,
+            max_tokens: 8192,
+            temperature: 0.7,
+            top_p: 0.9,
+            stream: false
+        };
+
+        let completionContent = null;
+
+        try {
+            console.log('Awaiting Hyperbolic API response...');
+            console.log('Messages:', messages);
+
+            const response = await fetch(this.apiUrl, {
+                method: 'POST',
+                headers: {
+                    'Content-Type': 'application/json',
+                    'Authorization': `Bearer ${this.apiKey}`
+                },
+                body: JSON.stringify(payload)
+            });
+
+            if (!response.ok) {
+                throw new Error(`HTTP error! status: ${response.status}`);
+            }
+
+            const data = await response.json();
+            if (
+                data?.choices?.[0]?.finish_reason &&
+                data.choices[0].finish_reason === 'length'
+            ) {
+                throw new Error('Context length exceeded');
+            }
+
+            completionContent = data?.choices?.[0]?.message?.content || '';
+            console.log('Received response from Hyperbolic.');
+
+        } catch (err) {
+            if (
+                (err.message === 'Context length exceeded' ||
+                 err.code === 'context_length_exceeded') &&
+                turns.length > 1
+            ) {
+                console.log('Context length exceeded, trying again with a shorter context...');
+                return await this.sendRequest(turns.slice(1), systemMessage, stopSeq);
+            } else {
+                console.log(err);
+                completionContent = 'My brain disconnected, try again.';
+            }
+        }
+        return completionContent.replace(/<\|separator\|>/g, '*no response*');
+    }
+    async embed(text) {
+        throw new Error('Embeddings are not supported by Hyperbolic.');
+    }
+}

src/models/local.js

@@ -0,0 +1,102 @@
+import { strictFormat } from '../utils/text.js';
+
+export class Local {
+    constructor(model_name, url, params) {
+        this.model_name = model_name;
+        this.params = params;
+        this.url = url || 'http://127.0.0.1:11434';
+        this.chat_endpoint = '/api/chat';
+        this.embedding_endpoint = '/api/embeddings';
+    }
+
+    async sendRequest(turns, systemMessage) {
+        let model = this.model_name || 'llama3';
+        let messages = strictFormat(turns);
+        messages.unshift({ role: 'system', content: systemMessage });
+        
+        // We'll attempt up to 5 times for models like "deepseek-r1" if the <think> tags are mismatched.
+        const maxAttempts = 5;
+        let attempt = 0;
+        let finalRes = null;
+
+        while (attempt < maxAttempts) {
+            attempt++;
+            console.log(`Awaiting local response... (model: ${model}, attempt: ${attempt})`);
+            let res = null;
+            try {
+                res = await this.send(this.chat_endpoint, {
+                    model: model,
+                    messages: messages,
+                    stream: false,
+                    ...(this.params || {})
+                });
+                if (res) {
+                    res = res['message']['content'];
+                } else {
+                    res = 'No response data.';
+                }
+            } catch (err) {
+                if (err.message.toLowerCase().includes('context length') && turns.length > 1) {
+                    console.log('Context length exceeded, trying again with shorter context.');
+                    return await this.sendRequest(turns.slice(1), systemMessage);
+                } else {
+                    console.log(err);
+                    res = 'My brain disconnected, try again.';
+                }
+            }
+
+            // If the model name includes "deepseek-r1" or "Andy-3.5-reasoning", then handle the <think> block.
+            if (this.model_name && this.model_name.includes("deepseek-r1") || this.model_name.includes("andy-3.5-reasoning")) {
+                const hasOpenTag = res.includes("<think>");
+                const hasCloseTag = res.includes("</think>");
+
+                // If there's a partial mismatch, retry to get a complete response.
+                if ((hasOpenTag && !hasCloseTag) || (!hasOpenTag && hasCloseTag)) {
+                    console.warn("Partial <think> block detected. Re-generating...");
+                    continue; 
+                }
+
+                // If both tags appear, remove them (and everything inside).
+                if (hasOpenTag && hasCloseTag) {
+                    res = res.replace(/<think>[\s\S]*?<\/think>/g, '');
+                }
+            }
+
+            finalRes = res;
+            break; // Exit the loop if we got a valid response.
+        }
+
+        if (finalRes == null) {
+            console.warn("Could not get a valid <think> block or normal response after max attempts.");
+            finalRes = 'Response incomplete, please try again.';
+        }
+        return finalRes;
+    }
+
+    async embed(text) {
+        let model = this.model_name || 'nomic-embed-text';
+        let body = { model: model, prompt: text };
+        let res = await this.send(this.embedding_endpoint, body);
+        return res['embedding'];
+    }
+
+    async send(endpoint, body) {
+        const url = new URL(endpoint, this.url);
+        let method = 'POST';
+        let headers = new Headers();
+        const request = new Request(url, { method, headers, body: JSON.stringify(body) });
+        let data = null;
+        try {
+            const res = await fetch(request);
+            if (res.ok) {
+                data = await res.json();
+            } else {
+                throw new Error(`Ollama Status: ${res.status}`);
+            }
+        } catch (err) {
+            console.error('Failed to send Ollama request.');
+            console.error(err);
+        }
+        return data;
+    }
+}

src/models/mistral.js

@@ -0,0 +1,73 @@
+import { Mistral as MistralClient } from '@mistralai/mistralai';
+import { getKey } from '../utils/keys.js';
+import { strictFormat } from '../utils/text.js';
+
+export class Mistral {
+    #client;
+
+    constructor(model_name, url, params) {
+        this.model_name = model_name;
+        this.params = params;
+
+        if (typeof url === "string") {
+            console.warn("Mistral does not support custom URL's, ignoring!");
+
+        }
+
+        if (!getKey("MISTRAL_API_KEY")) {
+            throw new Error("Mistral API Key missing, make sure to set MISTRAL_API_KEY in settings.json")
+        }
+
+        this.#client = new MistralClient(
+            {
+                apiKey: getKey("MISTRAL_API_KEY")
+            }
+        );
+
+        
+        // Prevents the following code from running when model not specified
+        if (typeof this.model_name === "undefined") return;
+
+        // get the model name without the "mistral" or "mistralai" prefix
+        // e.g "mistral/mistral-large-latest" -> "mistral-large-latest"
+        if (typeof model_name.split("/")[1] !== "undefined") {
+            this.model_name = model_name.split("/")[1];
+        }
+    }
+
+    async sendRequest(turns, systemMessage) {
+
+        let result;
+
+        try {
+            const model = this.model_name || "mistral-large-latest";
+
+            const messages = [
+                { role: "system", content: systemMessage }
+            ];
+            messages.push(...strictFormat(turns));
+
+            const response  = await this.#client.chat.complete({
+                model,
+                messages,
+                ...(this.params || {})
+            });
+
+            result = response.choices[0].message.content;
+        } catch (err) {
+            console.log(err)
+
+            result = "My brain disconnected, try again.";
+        }
+
+        return result;
+    }
+
+    async embed(text) {
+        const embedding = await this.#client.embeddings.create({
+            model: "mistral-embed",
+            inputs: text
+        });
+        return embedding.data[0].embedding;
+    }
+}

src/models/novita.js

@@ -0,0 +1,70 @@
+import OpenAIApi from 'openai';
+import { getKey } from '../utils/keys.js';
+import { strictFormat } from '../utils/text.js';
+
+// llama, mistral
+export class Novita {
+	constructor(model_name, url, params) {
+    this.model_name = model_name.replace('novita/', '');
+    this.url = url || 'https://api.novita.ai/v3/openai';
+    this.params = params;
+
+
+    let config = {
+      baseURL: this.url
+    };
+    config.apiKey = getKey('NOVITA_API_KEY');
+
+    this.openai = new OpenAIApi(config);
+  }
+
+	async sendRequest(turns, systemMessage, stop_seq='***') {
+      let messages = [{'role': 'system', 'content': systemMessage}].concat(turns);
+
+      
+      messages = strictFormat(messages);
+      
+      const pack = {
+          model: this.model_name || "meta-llama/llama-3.1-70b-instruct",
+          messages,
+          stop: [stop_seq],
+          ...(this.params || {})
+      };
+
+      let res = null;
+      try {
+          console.log('Awaiting novita api response...')
+          let completion = await this.openai.chat.completions.create(pack);
+          if (completion.choices[0].finish_reason == 'length')
+              throw new Error('Context length exceeded'); 
+          console.log('Received.')
+          res = completion.choices[0].message.content;
+      }
+      catch (err) {
+          if ((err.message == 'Context length exceeded' || err.code == 'context_length_exceeded') && turns.length > 1) {
+              console.log('Context length exceeded, trying again with shorter context.');
+              return await sendRequest(turns.slice(1), systemMessage, stop_seq);
+          } else {
+            console.log(err);
+              res = 'My brain disconnected, try again.';
+          }
+      }
+      if (res.includes('<think>')) {
+        let start = res.indexOf('<think>');
+        let end = res.indexOf('</think>') + 8;
+        if (start != -1) {
+          if (end != -1) {
+            res = res.substring(0, start) + res.substring(end);
+          } else {
+            res = res.substring(0, start+7);
+          }
+        }
+        res = res.trim();
+      }
+      return res;
+  }
+
+	async embed(text) {
+		throw new Error('Embeddings are not supported by Novita AI.');
+	}
+}

src/models/prompter.js

@@ -0,0 +1,373 @@
+import { readFileSync, mkdirSync, writeFileSync} from 'fs';
+import { Examples } from '../utils/examples.js';
+import { getCommandDocs } from '../agent/commands/index.js';
+import { getSkillDocs } from '../agent/library/index.js';
+import { SkillLibrary } from "../agent/library/skill_library.js";
+import { stringifyTurns } from '../utils/text.js';
+import { getCommand } from '../agent/commands/index.js';
+import settings from '../../settings.js';
+
+import { Gemini } from './gemini.js';
+import { GPT } from './gpt.js';
+import { Claude } from './claude.js';
+import { Mistral } from './mistral.js';
+import { ReplicateAPI } from './replicate.js';
+import { Local } from './local.js';
+import { Novita } from './novita.js';
+import { GroqCloudAPI } from './groq.js';
+import { HuggingFace } from './huggingface.js';
+import { Qwen } from "./qwen.js";
+import { Grok } from "./grok.js";
+import { DeepSeek } from './deepseek.js';
+import { hyperbolic } from './hyperbolic.js';
+import { glhf } from './glhf.js';
+
+export class Prompter {
+    constructor(agent, fp) {
+        this.agent = agent;
+        this.profile = JSON.parse(readFileSync(fp, 'utf8'));
+        let default_profile = JSON.parse(readFileSync('./profiles/defaults/_default.json', 'utf8'));
+        let base_fp = settings.base_profile;
+        let base_profile = JSON.parse(readFileSync(base_fp, 'utf8'));
+
+        // first use defaults to fill in missing values in the base profile
+        for (let key in default_profile) {
+            if (base_profile[key] === undefined)
+                base_profile[key] = default_profile[key];
+        }
+        // then use base profile to fill in missing values in the individual profile
+        for (let key in base_profile) {
+            if (this.profile[key] === undefined)
+                this.profile[key] = base_profile[key];
+        }
+        // base overrides default, individual overrides base
+        // Removed a bit of space that was right here by adding a comment instead of deleting it because I am making a pull request to this code and I can do whatever I want because I decided to add 2 new API services to Mindcraft now look at me go! Woohoo! I am flying off the edge of the screen oh no!
+
+        this.convo_examples = null;
+        this.coding_examples = null;
+        
+        let name = this.profile.name;
+        this.cooldown = this.profile.cooldown ? this.profile.cooldown : 0;
+        this.last_prompt_time = 0;
+        this.awaiting_coding = false;
+
+        // try to get "max_tokens" parameter, else null
+        let max_tokens = null;
+        if (this.profile.max_tokens)
+            max_tokens = this.profile.max_tokens;
+
+        let chat_model_profile = this._selectAPI(this.profile.model);
+        this.chat_model = this._createModel(chat_model_profile);
+
+        if (this.profile.code_model) {
+            let code_model_profile = this._selectAPI(this.profile.code_model);
+            this.code_model = this._createModel(code_model_profile);
+        }
+        else {
+            this.code_model = this.chat_model;
+        }
+
+        let embedding = this.profile.embedding;
+        if (embedding === undefined) {
+            if (chat_model_profile.api !== 'ollama')
+                embedding = {api: chat_model_profile.api};
+            else
+                embedding = {api: 'none'};
+        }
+        else if (typeof embedding === 'string' || embedding instanceof String)
+            embedding = {api: embedding};
+
+        console.log('Using embedding settings:', embedding);
+
+        try {
+            if (embedding.api === 'google')
+                this.embedding_model = new Gemini(embedding.model, embedding.url);
+            else if (embedding.api === 'openai')
+                this.embedding_model = new GPT(embedding.model, embedding.url);
+            else if (embedding.api === 'replicate')
+                this.embedding_model = new ReplicateAPI(embedding.model, embedding.url);
+            else if (embedding.api === 'ollama')
+                this.embedding_model = new Local(embedding.model, embedding.url);
+            else if (embedding.api === 'qwen')
+                this.embedding_model = new Qwen(embedding.model, embedding.url);
+            else if (embedding.api === 'mistral')
+                this.embedding_model = new Mistral(embedding.model, embedding.url);
+            else {
+                this.embedding_model = null;
+                console.log('Unknown embedding: ', embedding ? embedding.api : '[NOT SPECIFIED]', '. Using word overlap.');
+            }
+        }
+        catch (err) {
+            console.log('Warning: Failed to initialize embedding model:', err.message);
+            console.log('Continuing anyway, using word overlap instead.');
+            this.embedding_model = null;
+        }
+        this.skill_libary = new SkillLibrary(agent, this.embedding_model);
+        mkdirSync(`./bots/${name}`, { recursive: true });
+        writeFileSync(`./bots/${name}/last_profile.json`, JSON.stringify(this.profile, null, 4), (err) => {
+            if (err) {
+                throw new Error('Failed to save profile:', err);
+            }
+            console.log("Copy profile saved.");
+        });
+    }
+
+    _selectAPI(profile) {
+        if (typeof profile === 'string' || profile instanceof String) {
+            profile = {model: profile};
+        }
+        if (!profile.api) {
+            if (profile.model.includes('gemini'))
+                profile.api = 'google';
+            else if (profile.model.includes('gpt') || profile.model.includes('o1')|| profile.model.includes('o3'))
+                profile.api = 'openai';
+            else if (profile.model.includes('claude'))
+                profile.api = 'anthropic';
+            else if (profile.model.includes('huggingface/'))
+                profile.api = "huggingface";
+            else if (profile.model.includes('replicate/'))
+                profile.api = 'replicate';
+            else if (profile.model.includes('mistralai/') || profile.model.includes("mistral/"))
+                model_profile.api = 'mistral';
+            else if (profile.model.includes("groq/") || profile.model.includes("groqcloud/"))
+                profile.api = 'groq';
+            else if (chat.model.includes('hf:'))
+                chat.api = "glhf";
+            else if (chat.model.includes('hyperbolic:')|| chat.model.includes('hb:'))
+                chat.api = "hyperbolic";  
+            else if (profile.model.includes('novita/'))
+                profile.api = 'novita';
+            else if (profile.model.includes('qwen'))
+                profile.api = 'qwen';
+            else if (profile.model.includes('grok'))
+                profile.api = 'xai';
+            else if (profile.model.includes('deepseek'))
+                profile.api = 'deepseek';
+            else
+            profile.api = 'ollama';
+        }
+        return profile;
+    }
+
+    _createModel(profile) {
+        let model = null;
+        if (profile.api === 'google')
+            model = new Gemini(profile.model, profile.url, profile.params);
+        else if (profile.api === 'openai')
+            model = new GPT(profile.model, profile.url, profile.params);
+        else if (profile.api === 'anthropic')
+            model = new Claude(profile.model, profile.url, profile.params);
+        else if (profile.api === 'replicate')
+            model = new ReplicateAPI(profile.model, profile.url, profile.params);
+        else if (profile.api === 'ollama')
+            model = new Local(profile.model, profile.url, profile.params);
+        else if (profile.api === 'mistral')
+            model = new Mistral(profile.model, profile.url, profile.params);
+        else if (profile.api === 'groq')
+            model = new GroqCloudAPI(profile.model.replace('groq/', '').replace('groqcloud/', ''), profile.url, profile.params);
+        else if (profile.api === 'glhf')
+            model = new glhf(profile.model, profile.url, profile.params);
+        else if (profile.api === 'hyperbolic')
+            model = new hyperbolic(profile.model.replace('hyperbolic:', '').replace('hb:', ''), profile.url, profile.params); // Yes you can hate me for using curly braces on this little bit of code for defining the hyperbolic endpoint 
+        else if (profile.api === 'huggingface')
+            model = new HuggingFace(profile.model, profile.url, profile.params);
+        else if (profile.api === 'novita')
+            model = new Novita(profile.model.replace('novita/', ''), profile.url, profile.params);
+        else if (profile.api === 'qwen')
+            model = new Qwen(profile.model, profile.url, profile.params);
+        else if (profile.api === 'xai')
+            model = new Grok(profile.model, profile.url, profile.params);
+        else if (profile.api === 'deepseek')
+            model = new DeepSeek(profile.model, profile.url, profile.params);
+        else
+            throw new Error('Unknown API:', profile.api);
+        return model;
+    }
+
+    getName() {
+        return this.profile.name;
+    }
+
+    getInitModes() {
+        return this.profile.modes;
+    }
+
+    async initExamples() {
+        try {
+            this.convo_examples = new Examples(this.embedding_model, settings.num_examples);
+            this.coding_examples = new Examples(this.embedding_model, settings.num_examples);
+            
+            // Wait for both examples to load before proceeding
+            await Promise.all([
+                this.convo_examples.load(this.profile.conversation_examples),
+                this.coding_examples.load(this.profile.coding_examples),
+                this.skill_libary.initSkillLibrary()
+            ]);
+
+            console.log('Examples initialized.');
+        } catch (error) {
+            console.error('Failed to initialize examples:', error);
+            throw error;
+        }
+    }
+
+    async replaceStrings(prompt, messages, examples=null, to_summarize=[], last_goals=null) {
+        prompt = prompt.replaceAll('$NAME', this.agent.name);
+
+        if (prompt.includes('$STATS')) {
+            let stats = await getCommand('!stats').perform(this.agent);
+            prompt = prompt.replaceAll('$STATS', stats);
+        }
+        if (prompt.includes('$INVENTORY')) {
+            let inventory = await getCommand('!inventory').perform(this.agent);
+            prompt = prompt.replaceAll('$INVENTORY', inventory);
+        }
+        if (prompt.includes('$ACTION')) {
+            prompt = prompt.replaceAll('$ACTION', this.agent.actions.currentActionLabel);
+        }
+        if (prompt.includes('$COMMAND_DOCS'))
+            prompt = prompt.replaceAll('$COMMAND_DOCS', getCommandDocs());
+        if (prompt.includes('$CODE_DOCS')) {
+            const code_task_content = messages.slice().reverse().find(msg =>
+                msg.role !== 'system' && msg.content.includes('!newAction(')
+            )?.content?.match(/!newAction\((.*?)\)/)?.[1] || '';
+
+            prompt = prompt.replaceAll(
+                '$CODE_DOCS',
+                await this.skill_libary.getRelevantSkillDocs(code_task_content, settings.relevant_docs_count)
+            );
+        }
+            prompt = prompt.replaceAll('$COMMAND_DOCS', getCommandDocs());
+        if (prompt.includes('$CODE_DOCS'))
+            prompt = prompt.replaceAll('$CODE_DOCS', getSkillDocs());
+        if (prompt.includes('$EXAMPLES') && examples !== null)
+            prompt = prompt.replaceAll('$EXAMPLES', await examples.createExampleMessage(messages));
+        if (prompt.includes('$MEMORY'))
+            prompt = prompt.replaceAll('$MEMORY', this.agent.history.memory);
+        if (prompt.includes('$TO_SUMMARIZE'))
+            prompt = prompt.replaceAll('$TO_SUMMARIZE', stringifyTurns(to_summarize));
+        if (prompt.includes('$CONVO'))
+            prompt = prompt.replaceAll('$CONVO', 'Recent conversation:\n' + stringifyTurns(messages));
+        if (prompt.includes('$SELF_PROMPT')) {
+            let self_prompt = this.agent.self_prompter.on ? `YOUR CURRENT ASSIGNED GOAL: "${this.agent.self_prompter.prompt}"\n` : '';
+            prompt = prompt.replaceAll('$SELF_PROMPT', self_prompt);
+        }
+        if (prompt.includes('$LAST_GOALS')) {
+            let goal_text = '';
+            for (let goal in last_goals) {
+                if (last_goals[goal])
+                    goal_text += `You recently successfully completed the goal ${goal}.\n`
+                else
+                    goal_text += `You recently failed to complete the goal ${goal}.\n`
+            }
+            prompt = prompt.replaceAll('$LAST_GOALS', goal_text.trim());
+        }
+        if (prompt.includes('$BLUEPRINTS')) {
+            if (this.agent.npc.constructions) {
+                let blueprints = '';
+                for (let blueprint in this.agent.npc.constructions) {
+                    blueprints += blueprint + ', ';
+                }
+                prompt = prompt.replaceAll('$BLUEPRINTS', blueprints.slice(0, -2));
+            }
+        }
+
+        // check if there are any remaining placeholders with syntax $<word>
+        let remaining = prompt.match(/\$[A-Z_]+/g);
+        if (remaining !== null) {
+            console.warn('Unknown prompt placeholders:', remaining.join(', '));
+        }
+        return prompt;
+    }
+
+    async checkCooldown() {
+        let elapsed = Date.now() - this.last_prompt_time;
+        if (elapsed < this.cooldown && this.cooldown > 0) {
+            await new Promise(r => setTimeout(r, this.cooldown - elapsed));
+        }
+        this.last_prompt_time = Date.now();
+    }
+
+    async promptConvo(messages) {
+        this.most_recent_msg_time = Date.now();
+        let current_msg_time = this.most_recent_msg_time;
+        for (let i = 0; i < 3; i++) { // try 3 times to avoid hallucinations
+            await this.checkCooldown();
+            if (current_msg_time !== this.most_recent_msg_time) {
+                return '';
+            }
+            let prompt = this.profile.conversing;
+            prompt = await this.replaceStrings(prompt, messages, this.convo_examples);
+            let generation = await this.chat_model.sendRequest(messages, prompt);
+            // in conversations >2 players LLMs tend to hallucinate and role-play as other bots
+            // the FROM OTHER BOT tag should never be generated by the LLM
+            if (generation.includes('(FROM OTHER BOT)')) {
+                console.warn('LLM hallucinated message as another bot. Trying again...');
+                continue;
+            }
+            if (current_msg_time !== this.most_recent_msg_time) {
+                console.warn(this.agent.name + ' received new message while generating, discarding old response.');
+                return '';
+            }
+            return generation;
+        }
+        return '';
+    }
+
+    async promptCoding(messages) {
+        if (this.awaiting_coding) {
+            console.warn('Already awaiting coding response, returning no response.');
+            return '```//no response```';
+        }
+        this.awaiting_coding = true;
+        await this.checkCooldown();
+        let prompt = this.profile.coding;
+        prompt = await this.replaceStrings(prompt, messages, this.coding_examples);
+        let resp = await this.code_model.sendRequest(messages, prompt);
+        this.awaiting_coding = false;
+        return resp;
+    }
+
+    async promptMemSaving(to_summarize) {
+        await this.checkCooldown();
+        let prompt = this.profile.saving_memory;
+        prompt = await this.replaceStrings(prompt, null, null, to_summarize);
+        return await this.chat_model.sendRequest([], prompt);
+    }
+
+    async promptShouldRespondToBot(new_message) {
+        await this.checkCooldown();
+        let prompt = this.profile.bot_responder;
+        let messages = this.agent.history.getHistory();
+        messages.push({role: 'user', content: new_message});
+        prompt = await this.replaceStrings(prompt, null, null, messages);
+        let res = await this.chat_model.sendRequest([], prompt);
+        return res.trim().toLowerCase() === 'respond';
+    }
+
+    async promptGoalSetting(messages, last_goals) {
+        let system_message = this.profile.goal_setting;
+        system_message = await this.replaceStrings(system_message, messages);
+
+        let user_message = 'Use the below info to determine what goal to target next\n\n';
+        user_message += '$LAST_GOALS\n$STATS\n$INVENTORY\n$CONVO'
+        user_message = await this.replaceStrings(user_message, messages, null, null, last_goals);
+        let user_messages = [{role: 'user', content: user_message}];
+
+        let res = await this.chat_model.sendRequest(user_messages, system_message);
+
+        let goal = null;
+        try {
+            let data = res.split('```')[1].replace('json', '').trim();
+            goal = JSON.parse(data);
+        } catch (err) {
+            console.log('Failed to parse goal:', res, err);
+        }
+        if (!goal || !goal.name || !goal.quantity || isNaN(parseInt(goal.quantity))) {
+            console.log('Failed to set goal:', res);
+            return null;
+        }
+        goal.quantity = parseInt(goal.quantity);
+        return goal;
+    }
+}

src/models/qwen.js

@@ -0,0 +1,79 @@
+import OpenAIApi from 'openai';
+import { getKey, hasKey } from '../utils/keys.js';
+import { strictFormat } from '../utils/text.js';
+
+export class Qwen {
+    constructor(model_name, url, params) {
+        this.model_name = model_name;
+        this.params = params;
+        let config = {};
+
+        config.baseURL = url || 'https://dashscope.aliyuncs.com/compatible-mode/v1';
+        config.apiKey = getKey('QWEN_API_KEY');
+
+        this.openai = new OpenAIApi(config);
+    }
+
+    async sendRequest(turns, systemMessage, stop_seq='***') {
+        let messages = [{'role': 'system', 'content': systemMessage}].concat(turns);
+
+        messages = strictFormat(messages);
+
+        const pack = {
+            model: this.model_name || "qwen-plus",
+            messages,
+            stop: stop_seq,
+            ...(this.params || {})
+        };
+
+        let res = null;
+        try {
+            console.log('Awaiting Qwen api response...');
+            // console.log('Messages:', messages);
+            let completion = await this.openai.chat.completions.create(pack);
+            if (completion.choices[0].finish_reason == 'length')
+                throw new Error('Context length exceeded');
+            console.log('Received.');
+            res = completion.choices[0].message.content;
+        }
+        catch (err) {
+            if ((err.message == 'Context length exceeded' || err.code == 'context_length_exceeded') && turns.length > 1) {
+                console.log('Context length exceeded, trying again with shorter context.');
+                return await this.sendRequest(turns.slice(1), systemMessage, stop_seq);
+            } else {
+                console.log(err);
+                res = 'My brain disconnected, try again.';
+            }
+        }
+        return res;
+    }
+
+    // Why random backoff?
+    // With a 30 requests/second limit on Alibaba Qwen's embedding service,
+    // random backoff helps maximize bandwidth utilization.
+    async embed(text) {
+        const maxRetries = 5; // Maximum number of retries
+        for (let retries = 0; retries < maxRetries; retries++) {
+            try {
+                const { data } = await this.openai.embeddings.create({
+                    model: this.model_name || "text-embedding-v3",
+                    input: text,
+                    encoding_format: "float",
+                });
+                return data[0].embedding;
+            } catch (err) {
+                if (err.status === 429) {
+                    // If a rate limit error occurs, calculate the exponential backoff with a random delay (1-5 seconds)
+                    const delay = Math.pow(2, retries) * 1000 + Math.floor(Math.random() * 2000);
+                    // console.log(`Rate limit hit, retrying in ${delay} ms...`);
+                    await new Promise(resolve => setTimeout(resolve, delay)); // Wait for the delay before retrying
+                } else {
+                    throw err;
+                }
+            }
+        }
+        // If maximum retries are reached and the request still fails, throw an error
+        throw new Error('Max retries reached, request failed.');
+    }
+
+}

src/models/replicate.js

@@ -0,0 +1,59 @@
+import Replicate from 'replicate';
+import { toSinglePrompt } from '../utils/text.js';
+import { getKey } from '../utils/keys.js';
+
+// llama, mistral
+export class ReplicateAPI {
+	constructor(model_name, url, params) {
+		this.model_name = model_name;
+		this.url = url;
+		this.params = params;
+
+		if (this.url) {
+			console.warn('Replicate API does not support custom URLs. Ignoring provided URL.');
+		}
+
+		this.replicate = new Replicate({
+			auth: getKey('REPLICATE_API_KEY'),
+		});
+	}
+
+	async sendRequest(turns, systemMessage) {
+		const stop_seq = '***';
+		const prompt = toSinglePrompt(turns, null, stop_seq);
+		let model_name = this.model_name || 'meta/meta-llama-3-70b-instruct';
+
+		const input = { 
+			prompt, 
+			system_prompt: systemMessage,
+			...(this.params || {})
+		};
+		let res = null;
+		try {
+			console.log('Awaiting Replicate API response...');
+			let result = '';
+			for await (const event of this.replicate.stream(model_name, { input })) {
+				result += event;
+				if (result === '') break;
+				if (result.includes(stop_seq)) {
+					result = result.slice(0, result.indexOf(stop_seq));
+					break;
+				}
+			}
+			res = result;
+		} catch (err) {
+			console.log(err);
+			res = 'My brain disconnected, try again.';
+		}
+		console.log('Received.');
+		return res;
+	}
+
+	async embed(text) {
+		const output = await this.replicate.run(
+			this.model_name || "mark3labs/embeddings-gte-base:d619cff29338b9a37c3d06605042e1ff0594a8c3eff0175fd6967f5643fc4d47",
+			{ input: {text} }
+		);
+		return output.vectors;
+	}
+}

src/process/agent_process.js

@@ -0,0 +1,67 @@
+import { spawn } from 'child_process';
+import { mainProxy } from './main_proxy.js';
+
+export class AgentProcess {
+    start(profile, load_memory=false, init_message=null, count_id=0, task_path=null, task_id=null) {
+        this.profile = profile;
+        this.count_id = count_id;
+        this.running = true;
+
+        let args = ['src/process/init_agent.js', this.name];
+        args.push('-p', profile);
+        args.push('-c', count_id);
+        if (load_memory)
+            args.push('-l', load_memory);
+        if (init_message)
+            args.push('-m', init_message);
+        if (task_path)
+            args.push('-t', task_path);
+        if (task_id)
+            args.push('-i', task_id);
+
+        const agentProcess = spawn('node', args, {
+            stdio: 'inherit',
+            stderr: 'inherit',
+        });
+        
+        let last_restart = Date.now();
+        agentProcess.on('exit', (code, signal) => {
+            console.log(`Agent process exited with code ${code} and signal ${signal}`);
+            this.running = false;
+            mainProxy.logoutAgent(this.name);
+            
+            if (code > 1) {
+                console.log(`Ending task`);
+                process.exit(code);
+            }
+
+            if (code !== 0 && signal !== 'SIGINT') {
+                // agent must run for at least 10 seconds before restarting
+                if (Date.now() - last_restart < 10000) {
+                    console.error(`Agent process ${profile} exited too quickly and will not be restarted.`);
+                    return;
+                }
+                console.log('Restarting agent...');
+                this.start(profile, true, 'Agent process restarted.', count_id, task_path, task_id);
+                last_restart = Date.now();
+            }
+        });
+    
+        agentProcess.on('error', (err) => {
+            console.error('Agent process error:', err);
+        });
+
+        this.process = agentProcess;
+    }
+
+    stop() {
+        if (!this.running) return;
+        this.process.kill('SIGINT');
+    }
+
+    continue() {
+        if (!this.running) {
+            this.start(this.profile, true, 'Agent process restarted.', this.count_id);
+        }
+    }
+}

src/process/init_agent.js

@@ -0,0 +1,64 @@
+import { Agent } from '../agent/agent.js';
+import yargs from 'yargs';
+
+// Add global unhandled rejection handler
+process.on('unhandledRejection', (reason, promise) => {
+    console.error('Unhandled Rejection at:', {
+        promise: promise,
+        reason: reason,
+        stack: reason?.stack || 'No stack trace'
+    });
+    process.exit(1);
+});
+
+const args = process.argv.slice(2);
+if (args.length < 1) {
+    console.log('Usage: node init_agent.js <agent_name> [profile] [load_memory] [init_message]');
+    process.exit(1);
+}
+
+const argv = yargs(args)
+    .option('profile', {
+        alias: 'p',
+        type: 'string',
+        description: 'profile filepath to use for agent'
+    })
+    .option('load_memory', {
+        alias: 'l',
+        type: 'boolean',
+        description: 'load agent memory from file on startup'
+    })
+    .option('init_message', {
+        alias: 'm',
+        type: 'string',
+        description: 'automatically prompt the agent on startup'
+    })
+    .option('task_path', {
+        alias: 't',
+        type: 'string',
+        description: 'task filepath to use for agent'
+    })
+    .option('task_id', {
+        alias: 'i',
+        type: 'string',
+        description: 'task ID to execute'
+    })
+    .option('count_id', {
+        alias: 'c',
+        type: 'number',
+        default: 0,
+        description: 'identifying count for multi-agent scenarios',
+    }).argv;
+
+// Wrap agent start in async IIFE with proper error handling
+(async () => {
+    try {
+        console.log('Starting agent with profile:', argv.profile);
+        const agent = new Agent();
+        await agent.start(argv.profile, argv.load_memory, argv.init_message, argv.count_id, argv.task_path, argv.task_id);
+    } catch (error) {
+        console.error('Failed to start agent process:');
+        console.error(error);
+        process.exit(1);
+    }
+})();

src/process/main_proxy.js

@@ -0,0 +1,64 @@
+import { io } from 'socket.io-client';
+import settings from '../../settings.js';
+
+// Singleton mindserver proxy for the main process
+class MainProxy {
+    constructor() {
+        if (MainProxy.instance) {
+            return MainProxy.instance;
+        }
+        
+        this.socket = null;
+        this.connected = false;
+        this.agent_processes = {};
+        MainProxy.instance = this;
+    }
+
+    connect() {
+        if (this.connected) return;
+
+        this.socket = io(`http://${settings.mindserver_host}:${settings.mindserver_port}`);
+        this.connected = true;
+
+        this.socket.on('stop-agent', (agentName) => {
+            if (this.agent_processes[agentName]) {
+                this.agent_processes[agentName].stop();
+            }
+        });
+
+        this.socket.on('start-agent', (agentName) => {
+            if (this.agent_processes[agentName]) {
+                this.agent_processes[agentName].continue();
+            }
+        });
+
+        this.socket.on('register-agents-success', () => {
+            console.log('Agents registered');
+        });
+
+        this.socket.on('shutdown', () => {
+            console.log('Shutting down');
+            for (let agentName in this.agent_processes) {
+                this.agent_processes[agentName].stop();
+            }
+            setTimeout(() => {
+                process.exit(0);
+            }, 2000);
+        });
+    }
+
+    addAgent(agent) {
+        this.agent_processes.push(agent);
+    }
+
+    logoutAgent(agentName) {
+        this.socket.emit('logout-agent', agentName);
+    }
+
+    registerAgent(name, process) {
+        this.socket.emit('register-agents', [name]);
+        this.agent_processes[name] = process;
+    }
+}
+
+export const mainProxy = new MainProxy();

src/server/mind_server.js

@@ -0,0 +1,163 @@
+import { Server } from 'socket.io';
+import express from 'express';
+import http from 'http';
+import path from 'path';
+import { fileURLToPath } from 'url';
+
+// Module-level variables
+let io;
+let server;
+const registeredAgents = new Set();
+const inGameAgents = {};
+const agentManagers = {}; // socket for main process that registers/controls agents
+
+// Initialize the server
+export function createMindServer(port = 8080) {
+    const app = express();
+    server = http.createServer(app);
+    io = new Server(server);
+
+    // Serve static files
+    const __dirname = path.dirname(fileURLToPath(import.meta.url));
+    app.use(express.static(path.join(__dirname, 'public')));
+
+    // Socket.io connection handling
+    io.on('connection', (socket) => {
+        let curAgentName = null;
+        console.log('Client connected');
+
+        agentsUpdate(socket);
+
+        socket.on('register-agents', (agentNames) => {
+            console.log(`Registering agents: ${agentNames}`);
+            agentNames.forEach(name => registeredAgents.add(name));
+            for (let name of agentNames) {
+                agentManagers[name] = socket;
+            }
+            socket.emit('register-agents-success');
+            agentsUpdate();
+        });
+
+        socket.on('login-agent', (agentName) => {
+            if (curAgentName && curAgentName !== agentName) {
+                console.warn(`Agent ${agentName} already logged in as ${curAgentName}`);
+                return;
+            }
+            if (registeredAgents.has(agentName)) {
+                curAgentName = agentName;
+                inGameAgents[agentName] = socket;
+                agentsUpdate();
+            } else {
+                console.warn(`Agent ${agentName} not registered`);
+            }
+        });
+
+        socket.on('logout-agent', (agentName) => {
+            if (inGameAgents[agentName]) {
+                delete inGameAgents[agentName];
+                agentsUpdate();
+            }
+        });
+
+        socket.on('disconnect', () => {
+            console.log('Client disconnected');
+            if (inGameAgents[curAgentName]) {
+                delete inGameAgents[curAgentName];
+                agentsUpdate();
+            }
+        });
+
+        socket.on('chat-message', (agentName, json) => {
+            if (!inGameAgents[agentName]) {
+                console.warn(`Agent ${agentName} tried to send a message but is not logged in`);
+                return;
+            }
+            console.log(`${curAgentName} sending message to ${agentName}: ${json.message}`);
+            inGameAgents[agentName].emit('chat-message', curAgentName, json);
+        });
+
+        socket.on('restart-agent', (agentName) => {
+            console.log(`Restarting agent: ${agentName}`);
+            inGameAgents[agentName].emit('restart-agent');
+        });
+
+        socket.on('stop-agent', (agentName) => {
+            let manager = agentManagers[agentName];
+            if (manager) {
+                manager.emit('stop-agent', agentName);
+            }
+            else {
+                console.warn(`Stopping unregisterd agent ${agentName}`);
+            }
+        });
+
+        socket.on('start-agent', (agentName) => {
+            let manager = agentManagers[agentName];
+            if (manager) {
+                manager.emit('start-agent', agentName);
+            }
+            else {
+                console.warn(`Starting unregisterd agent ${agentName}`);
+            }
+        });
+
+        socket.on('stop-all-agents', () => {
+            console.log('Killing all agents');
+            stopAllAgents();
+        });
+
+        socket.on('shutdown', () => {
+            console.log('Shutting down');
+            for (let manager of Object.values(agentManagers)) {
+                manager.emit('shutdown');
+            }
+            setTimeout(() => {
+                process.exit(0);
+            }, 2000);
+        });
+
+		socket.on('send-message', (agentName, message) => {
+			if (!inGameAgents[agentName]) {
+				console.warn(`Agent ${agentName} not logged in, cannot send message via MindServer.`);
+				return
+			}
+			try {
+				console.log(`Sending message to agent ${agentName}: ${message}`);
+				inGameAgents[agentName].emit('send-message', agentName, message)
+			} catch (error) {
+				console.error('Error: ', error);
+			}
+		});
+    });
+
+    server.listen(port, 'localhost', () => {
+        console.log(`MindServer running on port ${port}`);
+    });
+
+    return server;
+}
+
+function agentsUpdate(socket) {
+    if (!socket) {
+        socket = io;
+    }
+    let agents = [];
+    registeredAgents.forEach(name => {
+        agents.push({name, in_game: !!inGameAgents[name]});
+    });
+    socket.emit('agents-update', agents);
+}
+
+function stopAllAgents() {
+    for (const agentName in inGameAgents) {
+        let manager = agentManagers[agentName];
+        if (manager) {
+            manager.emit('stop-agent', agentName);
+        }
+    }
+}
+
+// Optional: export these if you need access to them from other files
+export const getIO = () => io;
+export const getServer = () => server;
+export const getConnectedAgents = () => connectedAgents; 

src/server/public/index.html

@@ -0,0 +1,120 @@
+<!DOCTYPE html>
+<html>
+<head>
+    <title>Mindcraft</title>
+    <script src="/socket.io/socket.io.js"></script>
+    <style>
+        body {
+            font-family: Arial, sans-serif;
+            margin: 20px;
+            background: #1a1a1a;
+            color: #e0e0e0;
+        }
+        #agents {
+            background: #2d2d2d;
+            padding: 20px;
+            border-radius: 8px;
+            box-shadow: 0 2px 4px rgba(0,0,0,0.2);
+        }
+        h1 {
+            color: #ffffff;
+        }
+        .agent {
+            margin: 10px 0;
+            padding: 10px;
+            background: #363636;
+            border-radius: 4px;
+            display: flex;
+            justify-content: space-between;
+            align-items: center;
+        }
+        .restart-btn, .start-btn, .stop-btn {
+            color: white;
+            border: none;
+            padding: 5px 10px;
+            border-radius: 4px;
+            cursor: pointer;
+            margin-left: 5px;
+        }
+        .restart-btn {
+            background: #4CAF50;
+        }
+        .start-btn {
+            background: #2196F3;
+        }
+        .stop-btn {
+            background: #f44336;
+        }
+        .restart-btn:hover { background: #45a049; }
+        .start-btn:hover { background: #1976D2; }
+        .stop-btn:hover { background: #d32f2f; }
+        .status-icon {
+            font-size: 12px;
+            margin-right: 8px;
+        }
+        .status-icon.online {
+            color: #4CAF50;
+        }
+        .status-icon.offline {
+            color: #f44336;
+        }
+    </style>
+</head>
+<body>
+    <h1>Mindcraft</h1>
+    <div id="agents"></div>
+
+    <script>
+        const socket = io();
+        const agentsDiv = document.getElementById('agents');
+
+        socket.on('agents-update', (agents) => {
+            agentsDiv.innerHTML = agents.length ? 
+                agents.map(agent => `
+                    <div class="agent">
+                        <span>
+                            <span class="status-icon ${agent.in_game ? 'online' : 'offline'}">●</span>
+                            ${agent.name}
+                        </span>
+                        <div>
+                            ${agent.in_game ? `
+                                <button class="stop-btn" onclick="stopAgent('${agent.name}')">Stop</button>
+                                <button class="restart-btn" onclick="restartAgent('${agent.name}')">Restart</button>
+								<input type="text" id="messageInput" placeholder="Enter a message or command..."></input><button class="start-btn" onclick="sendMessage('${agent.name}', document.getElementById('messageInput').value)">Send</button>
+                            ` : `
+                                <button class="start-btn" onclick="startAgent('${agent.name}')">Start</button>
+                            `}
+                        </div>
+                    </div>
+                `).join('') + 
+                `<button class="stop-btn" onclick="killAllAgents()">Stop All</button>
+                <button class="stop-btn" onclick="shutdown()">Shutdown</button>` :
+                '<div class="agent">No agents connected</div>';
+        });
+
+        function restartAgent(agentName) {
+            socket.emit('restart-agent', agentName);
+        }
+
+        function startAgent(agentName) {
+            socket.emit('start-agent', agentName);
+        }
+
+        function stopAgent(agentName) {
+            socket.emit('stop-agent', agentName);
+        }
+
+        function killAllAgents() {
+            socket.emit('stop-all-agents');
+        }
+
+        function shutdown() {
+            socket.emit('shutdown');
+        }
+		
+		function sendMessage(agentName, message) {
+			socket.emit('send-message', agentName, message)
+		}
+    </script>
+</body>
+</html> 

src/utils/examples.js

@@ -0,0 +1,94 @@
+import { cosineSimilarity } from './math.js';
+import { stringifyTurns } from './text.js';
+
+export class Examples {
+    constructor(model, select_num=2) {
+        this.examples = [];
+        this.model = model;
+        this.select_num = select_num;
+        this.embeddings = {};
+    }
+
+    turnsToText(turns) {
+        let messages = '';
+        for (let turn of turns) {
+            if (turn.role !== 'assistant')
+                messages += turn.content.substring(turn.content.indexOf(':')+1).trim() + '\n';
+        }
+        return messages.trim();
+    }
+
+    getWords(text) {
+        return text.replace(/[^a-zA-Z ]/g, '').toLowerCase().split(' ');
+    }
+
+    wordOverlapScore(text1, text2) {
+        const words1 = this.getWords(text1);
+        const words2 = this.getWords(text2);
+        const intersection = words1.filter(word => words2.includes(word));
+        return intersection.length / (words1.length + words2.length - intersection.length);
+    }
+
+    async load(examples) {
+        this.examples = examples;
+        if (!this.model) return; // Early return if no embedding model
+        
+        if (this.select_num === 0)
+            return;
+
+        try {
+            // Create array of promises first
+            const embeddingPromises = examples.map(example => {
+                const turn_text = this.turnsToText(example);
+                return this.model.embed(turn_text)
+                    .then(embedding => {
+                        this.embeddings[turn_text] = embedding;
+                    });
+            });
+            
+            // Wait for all embeddings to complete
+            await Promise.all(embeddingPromises);
+        } catch (err) {
+            console.warn('Error with embedding model, using word overlap instead:', err);
+            this.model = null;
+        }
+    }
+
+    async getRelevant(turns) {
+        if (this.select_num === 0)
+            return [];
+
+        let turn_text = this.turnsToText(turns);
+        if (this.model !== null) {
+            let embedding = await this.model.embed(turn_text);
+            this.examples.sort((a, b) => 
+                cosineSimilarity(embedding, this.embeddings[this.turnsToText(b)]) -
+                cosineSimilarity(embedding, this.embeddings[this.turnsToText(a)])
+            );
+        }
+        else {
+            this.examples.sort((a, b) => 
+                this.wordOverlapScore(turn_text, this.turnsToText(b)) -
+                this.wordOverlapScore(turn_text, this.turnsToText(a))
+            );
+        }
+        let selected = this.examples.slice(0, this.select_num);
+        return JSON.parse(JSON.stringify(selected)); // deep copy
+    }
+
+    async createExampleMessage(turns) {
+        let selected_examples = await this.getRelevant(turns);
+
+        console.log('selected examples:');
+        for (let example of selected_examples) {
+            console.log('Example:', example[0].content)
+        }
+
+        let msg = 'Examples of how to respond:\n';
+        for (let i=0; i<selected_examples.length; i++) {
+            let example = selected_examples[i];
+            msg += `Example ${i+1}:\n${stringifyTurns(example)}\n\n`;
+        }
+        return msg;
+    }
+}

src/utils/keys.js

@@ -0,0 +1,24 @@
+import { readFileSync } from 'fs';
+
+let keys = {};
+try {
+    const data = readFileSync('./keys.json', 'utf8');
+    keys = JSON.parse(data);
+} catch (err) {
+    console.warn('keys.json not found. Defaulting to environment variables.'); // still works with local models
+}
+
+export function getKey(name) {
+    let key = keys[name];
+    if (!key) {
+        key = process.env[name];
+    }
+    if (!key) {
+        throw new Error(`API key "${name}" not found in keys.json or environment variables!`);
+    }
+    return keys[name];
+}
+
+export function hasKey(name) {
+    return keys[name] || process.env[name];
+}

src/utils/math.js

@@ -0,0 +1,13 @@
+export function cosineSimilarity(a, b) {
+    let dotProduct = 0;
+    let magnitudeA = 0;
+    let magnitudeB = 0;
+    for (let i = 0; i < a.length; i++) {
+        dotProduct += a[i] * b[i];  // calculate dot product
+        magnitudeA += Math.pow(a[i], 2);  // calculate magnitude of a
+        magnitudeB += Math.pow(b[i], 2);  // calculate magnitude of b
+    }
+    magnitudeA = Math.sqrt(magnitudeA);
+    magnitudeB = Math.sqrt(magnitudeB);
+    return dotProduct / (magnitudeA * magnitudeB);  // calculate cosine similarity
+}

src/utils/mcdata.js

@@ -0,0 +1,485 @@
+import minecraftData from 'minecraft-data';
+import settings from '../../settings.js';
+import { createBot } from 'mineflayer';
+import prismarine_items from 'prismarine-item';
+import { pathfinder } from 'mineflayer-pathfinder';
+import { plugin as pvp } from 'mineflayer-pvp';
+import { plugin as collectblock } from 'mineflayer-collectblock';
+import { plugin as autoEat } from 'mineflayer-auto-eat';
+import plugin from 'mineflayer-armor-manager';
+const armorManager = plugin;
+
+const mc_version = settings.minecraft_version;
+const mcdata = minecraftData(mc_version);
+const Item = prismarine_items(mc_version);
+
+/**
+ * @typedef {string} ItemName
+ * @typedef {string} BlockName
+*/
+
+export const WOOD_TYPES = ['oak', 'spruce', 'birch', 'jungle', 'acacia', 'dark_oak', 'mangrove', 'cherry'];
+export const MATCHING_WOOD_BLOCKS = [
+    'log',
+    'planks',
+    'sign',
+    'boat',
+    'fence_gate',
+    'door',
+    'fence',
+    'slab',
+    'stairs',
+    'button',
+    'pressure_plate',
+    'trapdoor'
+]
+export const WOOL_COLORS = [
+    'white',
+    'orange',
+    'magenta',
+    'light_blue',
+    'yellow',
+    'lime',
+    'pink',
+    'gray',
+    'light_gray',
+    'cyan',
+    'purple',
+    'blue',
+    'brown',
+    'green',
+    'red',
+    'black'
+]
+
+
+export function initBot(username) {
+    let bot = createBot({
+        username: username,
+
+        host: settings.host,
+        port: settings.port,
+        auth: settings.auth,
+
+        version: mc_version,
+    });
+    bot.loadPlugin(pathfinder);
+    bot.loadPlugin(pvp);
+    bot.loadPlugin(collectblock);
+    bot.loadPlugin(autoEat);
+    bot.loadPlugin(armorManager); // auto equip armor
+    bot.once('resourcePack', () => {
+        bot.acceptResourcePack();
+    });
+
+    return bot;
+}
+
+export function isHuntable(mob) {
+    if (!mob || !mob.name) return false;
+    const animals = ['chicken', 'cow', 'llama', 'mooshroom', 'pig', 'rabbit', 'sheep'];
+    return animals.includes(mob.name.toLowerCase()) && !mob.metadata[16]; // metadata 16 is not baby
+}
+
+export function isHostile(mob) {
+    if (!mob || !mob.name) return false;
+    return  (mob.type === 'mob' || mob.type === 'hostile') && mob.name !== 'iron_golem' && mob.name !== 'snow_golem';
+}
+
+export function getItemId(itemName) {
+    let item = mcdata.itemsByName[itemName];
+    if (item) {
+        return item.id;
+    }
+    return null;
+}
+
+export function getItemName(itemId) {
+    let item = mcdata.items[itemId]
+    if (item) {
+        return item.name;
+    }
+    return null;
+}
+
+export function getBlockId(blockName) {
+    let block = mcdata.blocksByName[blockName];
+    if (block) {
+        return block.id;
+    }
+    return null;
+}
+
+export function getBlockName(blockId) {
+    let block = mcdata.blocks[blockId]
+    if (block) {
+        return block.name;
+    }
+    return null;
+}
+
+export function getAllItems(ignore) {
+    if (!ignore) {
+        ignore = [];
+    }
+    let items = []
+    for (const itemId in mcdata.items) {
+        const item = mcdata.items[itemId];
+        if (!ignore.includes(item.name)) {
+            items.push(item);
+        }
+    }
+    return items;
+}
+
+export function getAllItemIds(ignore) {
+    const items = getAllItems(ignore);
+    let itemIds = [];
+    for (const item of items) {
+        itemIds.push(item.id);
+    }
+    return itemIds;
+}
+
+export function getAllBlocks(ignore) {
+    if (!ignore) {
+        ignore = [];
+    }
+    let blocks = []
+    for (const blockId in mcdata.blocks) {
+        const block = mcdata.blocks[blockId];
+        if (!ignore.includes(block.name)) {
+            blocks.push(block);
+        }
+    }
+    return blocks;
+}
+
+export function getAllBlockIds(ignore) {
+    const blocks = getAllBlocks(ignore);
+    let blockIds = [];
+    for (const block of blocks) {
+        blockIds.push(block.id);
+    }
+    return blockIds;
+}
+
+export function getAllBiomes() {
+    return mcdata.biomes;
+}
+
+export function getItemCraftingRecipes(itemName) {
+    let itemId = getItemId(itemName);
+    if (!mcdata.recipes[itemId]) {
+        return null;
+    }
+
+    let recipes = [];
+    for (let r of mcdata.recipes[itemId]) {
+        let recipe = {};
+        let ingredients = [];
+        if (r.ingredients) {
+            ingredients = r.ingredients;
+        } else if (r.inShape) {
+            ingredients = r.inShape.flat();
+        }
+        for (let ingredient of ingredients) {
+            let ingredientName = getItemName(ingredient);
+            if (ingredientName === null) continue;
+            if (!recipe[ingredientName])
+                recipe[ingredientName] = 0;
+            recipe[ingredientName]++;
+        }
+        recipes.push([
+            recipe,
+            {craftedCount : r.result.count}
+        ]);
+    }
+
+    return recipes;
+}
+
+export function isSmeltable(itemName) {
+    const misc_smeltables = ['beef', 'chicken', 'cod', 'mutton', 'porkchop', 'rabbit', 'salmon', 'tropical_fish', 'potato', 'kelp', 'sand', 'cobblestone', 'clay_ball'];
+    return itemName.includes('raw') || itemName.includes('log') || misc_smeltables.includes(itemName);
+}
+
+export function getSmeltingFuel(bot) {
+    let fuel = bot.inventory.items().find(i => i.name === 'coal' || i.name === 'charcoal' || i.name === 'blaze_rod')
+    if (fuel)
+        return fuel;
+    fuel = bot.inventory.items().find(i => i.name.includes('log') || i.name.includes('planks'))
+    if (fuel)
+        return fuel;
+    return bot.inventory.items().find(i => i.name === 'coal_block' || i.name === 'lava_bucket');
+}
+
+export function getFuelSmeltOutput(fuelName) {
+    if (fuelName === 'coal' || fuelName === 'charcoal')
+        return 8;
+    if (fuelName === 'blaze_rod')
+        return 12;
+    if (fuelName.includes('log') || fuelName.includes('planks'))
+        return 1.5
+    if (fuelName === 'coal_block')
+        return 80;
+    if (fuelName === 'lava_bucket')
+        return 100;
+    return 0;
+}
+
+export function getItemSmeltingIngredient(itemName) {
+    return {    
+        baked_potato: 'potato',
+        steak: 'raw_beef',
+        cooked_chicken: 'raw_chicken',
+        cooked_cod: 'raw_cod',
+        cooked_mutton: 'raw_mutton',
+        cooked_porkchop: 'raw_porkchop',
+        cooked_rabbit: 'raw_rabbit',
+        cooked_salmon: 'raw_salmon',
+        dried_kelp: 'kelp',
+        iron_ingot: 'raw_iron',
+        gold_ingot: 'raw_gold',
+        copper_ingot: 'raw_copper',
+        glass: 'sand'
+    }[itemName];
+}
+
+export function getItemBlockSources(itemName) {
+    let itemId = getItemId(itemName);
+    let sources = [];
+    for (let block of getAllBlocks()) {
+        if (block.drops.includes(itemId)) {
+            sources.push(block.name);
+        }
+    }
+    return sources;
+}
+
+export function getItemAnimalSource(itemName) {
+    return {    
+        raw_beef: 'cow',
+        raw_chicken: 'chicken',
+        raw_cod: 'cod',
+        raw_mutton: 'sheep',
+        raw_porkchop: 'pig',
+        raw_rabbit: 'rabbit',
+        raw_salmon: 'salmon',
+        leather: 'cow',
+        wool: 'sheep'
+    }[itemName];
+}
+
+export function getBlockTool(blockName) {
+    let block = mcdata.blocksByName[blockName];
+    if (!block || !block.harvestTools) {
+        return null;
+    }
+    return getItemName(Object.keys(block.harvestTools)[0]);  // Double check first tool is always simplest
+}
+
+export function makeItem(name, amount=1) {
+    return new Item(getItemId(name), amount);
+}
+
+/**
+ * Returns the number of ingredients required to use the recipe once.
+ * 
+ * @param {Recipe} recipe
+ * @returns {Object<mc.ItemName, number>} an object describing the number of each ingredient.
+ */
+export function ingredientsFromPrismarineRecipe(recipe) {
+    let requiredIngedients = {};
+    if (recipe.inShape)
+        for (const ingredient of recipe.inShape.flat()) {
+            if(ingredient.id<0) continue; //prismarine-recipe uses id -1 as an empty crafting slot
+            const ingredientName = getItemName(ingredient.id);
+            requiredIngedients[ingredientName] ??=0;
+            requiredIngedients[ingredientName] += ingredient.count;
+        }
+    if (recipe.ingredients)
+        for (const ingredient of recipe.ingredients) {
+            if(ingredient.id<0) continue;
+            const ingredientName = getItemName(ingredient.id);
+            requiredIngedients[ingredientName] ??=0;
+            requiredIngedients[ingredientName] -= ingredient.count;
+            //Yes, the `-=` is intended.
+            //prismarine-recipe uses positive numbers for the shaped ingredients but negative for unshaped.
+            //Why this is the case is beyond my understanding.
+        }
+    return requiredIngedients;
+}
+
+/**
+ * Calculates the number of times an action, such as a crafing recipe, can be completed before running out of resources.
+ * @template T - doesn't have to be an item. This could be any resource.
+ * @param {Object.<T, number>} availableItems - The resources available; e.g, `{'cobble_stone': 7, 'stick': 10}`
+ * @param {Object.<T, number>} requiredItems - The resources required to complete the action once; e.g, `{'cobble_stone': 3, 'stick': 2}`
+ * @param {boolean} discrete - Is the action discrete?
+ * @returns {{num: number, limitingResource: (T | null)}} the number of times the action can be completed and the limmiting resource; e.g `{num: 2, limitingResource: 'cobble_stone'}`
+ */
+export function calculateLimitingResource(availableItems, requiredItems, discrete=true) {
+    let limitingResource = null;
+    let num = Infinity;
+    for (const itemType in requiredItems) {
+        if (availableItems[itemType] < requiredItems[itemType] * num) {
+            limitingResource = itemType;
+            num = availableItems[itemType] / requiredItems[itemType];
+        }
+    }
+    if(discrete) num = Math.floor(num);
+    return {num, limitingResource}
+}
+
+let loopingItems = new Set();
+
+export function initializeLoopingItems() {
+
+    loopingItems = new Set(['coal',
+        'wheat',
+        'diamond',
+        'emerald',
+        'raw_iron',
+        'raw_gold',
+        'redstone',
+        'blue_wool',
+        'packed_mud',
+        'raw_copper',
+        'iron_ingot',
+        'dried_kelp',
+        'gold_ingot',
+        'slime_ball',
+        'black_wool',
+        'quartz_slab',
+        'copper_ingot',
+        'lapis_lazuli',
+        'honey_bottle',
+        'rib_armor_trim_smithing_template',
+        'eye_armor_trim_smithing_template',
+        'vex_armor_trim_smithing_template',
+        'dune_armor_trim_smithing_template',
+        'host_armor_trim_smithing_template',
+        'tide_armor_trim_smithing_template',
+        'wild_armor_trim_smithing_template',
+        'ward_armor_trim_smithing_template',
+        'coast_armor_trim_smithing_template',
+        'spire_armor_trim_smithing_template',
+        'snout_armor_trim_smithing_template',
+        'shaper_armor_trim_smithing_template',
+        'netherite_upgrade_smithing_template',
+        'raiser_armor_trim_smithing_template',
+        'sentry_armor_trim_smithing_template',
+        'silence_armor_trim_smithing_template',
+        'wayfinder_armor_trim_smithing_template']);
+}
+
+
+/**
+ * Gets a detailed plan for crafting an item considering current inventory
+ */
+export function getDetailedCraftingPlan(targetItem, count = 1, current_inventory = {}) {
+    initializeLoopingItems();
+    if (!targetItem || count <= 0 || !getItemId(targetItem)) {
+        return "Invalid input. Please provide a valid item name and positive count.";
+    }
+
+    if (isBaseItem(targetItem)) {
+        const available = current_inventory[targetItem] || 0;
+        if (available >= count) return "You have all required items already in your inventory!";
+        return `${targetItem} is a base item, you need to find ${count - available} more in the world`;
+    }
+
+    const inventory = { ...current_inventory };
+    const leftovers = {};
+    const plan = craftItem(targetItem, count, inventory, leftovers);
+    return formatPlan(plan);
+}
+
+function isBaseItem(item) {
+    return loopingItems.has(item) || getItemCraftingRecipes(item) === null;
+}
+
+function craftItem(item, count, inventory, leftovers, crafted = { required: {}, steps: [], leftovers: {} }) {
+    // Check available inventory and leftovers first
+    const availableInv = inventory[item] || 0;
+    const availableLeft = leftovers[item] || 0;
+    const totalAvailable = availableInv + availableLeft;
+
+    if (totalAvailable >= count) {
+        // Use leftovers first, then inventory
+        const useFromLeft = Math.min(availableLeft, count);
+        leftovers[item] = availableLeft - useFromLeft;
+        
+        const remainingNeeded = count - useFromLeft;
+        if (remainingNeeded > 0) {
+            inventory[item] = availableInv - remainingNeeded;
+        }
+        return crafted;
+    }
+
+    // Use whatever is available
+    const stillNeeded = count - totalAvailable;
+    if (availableLeft > 0) leftovers[item] = 0;
+    if (availableInv > 0) inventory[item] = 0;
+
+    if (isBaseItem(item)) {
+        crafted.required[item] = (crafted.required[item] || 0) + stillNeeded;
+        return crafted;
+    }
+
+    const recipe = getItemCraftingRecipes(item)?.[0];
+    if (!recipe) {
+        crafted.required[item] = stillNeeded;
+        return crafted;
+    }
+
+    const [ingredients, result] = recipe;
+    const craftedPerRecipe = result.craftedCount;
+    const batchCount = Math.ceil(stillNeeded / craftedPerRecipe);
+    const totalProduced = batchCount * craftedPerRecipe;
+
+    // Add excess to leftovers
+    if (totalProduced > stillNeeded) {
+        leftovers[item] = (leftovers[item] || 0) + (totalProduced - stillNeeded);
+    }
+
+    // Process each ingredient
+    for (const [ingredientName, ingredientCount] of Object.entries(ingredients)) {
+        const totalIngredientNeeded = ingredientCount * batchCount;
+        craftItem(ingredientName, totalIngredientNeeded, inventory, leftovers, crafted);
+    }
+
+    // Add crafting step
+    const stepIngredients = Object.entries(ingredients)
+        .map(([name, amount]) => `${amount * batchCount} ${name}`)
+        .join(' + ');
+    crafted.steps.push(`Craft ${stepIngredients} -> ${totalProduced} ${item}`);
+
+    return crafted;
+}
+
+function formatPlan({ required, steps, leftovers }) {
+    const lines = [];
+
+    if (Object.keys(required).length > 0) {
+        lines.push('You are missing the following items:');
+        Object.entries(required).forEach(([item, count]) => 
+            lines.push(`- ${count} ${item}`));
+        lines.push('\nOnce you have these items, here\'s your crafting plan:');
+    } else {
+        lines.push('You have all items required to craft this item!');
+        lines.push('Here\'s your crafting plan:');
+    }
+
+    lines.push('');
+    lines.push(...steps);
+
+    if (Object.keys(leftovers).length > 0) {
+        lines.push('\nYou will have leftover:');
+        Object.entries(leftovers).forEach(([item, count]) => 
+            lines.push(`- ${count} ${item}`));
+    }
+
+    return lines.join('\n');
+}

src/utils/text.js

@@ -0,0 +1,65 @@
+export function stringifyTurns(turns) {
+    let res = '';
+    for (let turn of turns) {
+        if (turn.role === 'assistant') {
+            res += `\nYour output:\n${turn.content}`;
+        } else if (turn.role === 'system') {
+            res += `\nSystem output: ${turn.content}`;
+        } else {
+            res += `\nUser input: ${turn.content}`;
+        
+        }
+    }
+    return res.trim();
+}
+
+export function toSinglePrompt(turns, system=null, stop_seq='***', model_nickname='assistant') {
+    let prompt = system ? `${system}${stop_seq}` : '';
+    let role = '';
+    turns.forEach((message) => {
+        role = message.role;
+        if (role === 'assistant') role = model_nickname;
+        prompt += `${role}: ${message.content}${stop_seq}`;
+    });
+    if (role !== model_nickname) // if the last message was from the user/system, add a prompt for the model. otherwise, pretend we are extending the model's own message
+        prompt += model_nickname + ": ";
+    return prompt;
+}
+
+// ensures stricter turn order and roles:
+// - system messages are treated as user messages and prefixed with SYSTEM:
+// - combines repeated messages from users
+// - separates repeat assistant messages with filler user messages
+export function strictFormat(turns) {
+    let prev_role = null;
+    let messages = [];
+    let filler = {role: 'user', content: '_'};
+    for (let msg of turns) {
+        msg.content = msg.content.trim();
+        if (msg.role === 'system') {
+            msg.role = 'user';
+            msg.content = 'SYSTEM: ' + msg.content;
+        }
+        if (msg.role === prev_role && msg.role === 'assistant') {
+            // insert empty user message to separate assistant messages
+            messages.push(filler);
+            messages.push(msg);
+        }
+        else if (msg.role === prev_role) {
+            // combine new message with previous message instead of adding a new one
+            messages[messages.length-1].content += '\n' + msg.content;
+        }
+        else {
+            messages.push(msg);
+        }
+        prev_role = msg.role;
+        
+    }
+    if (messages.length > 0 && messages[0].role !== 'user') {
+        messages.unshift(filler); // anthropic requires user message to start
+    }
+    if (messages.length === 0) {
+        messages.push(filler);
+    }
+    return messages;
+}

src/utils/translator.js

@@ -0,0 +1,28 @@
+import translate from 'google-translate-api-x';
+import settings from '../../settings.js';
+
+const preferred_lang = String(settings.language).toLowerCase();
+
+export async function handleTranslation(message) {
+    if (preferred_lang === 'en' || preferred_lang === 'english')
+        return message;
+    try {
+        const translation = await translate(message, { to: preferred_lang });
+        return translation.text || message;
+    } catch (error) {
+        console.error('Error translating message:', error);
+        return message;
+    }
+}
+
+export async function handleEnglishTranslation(message) {
+    if (preferred_lang === 'en' || preferred_lang === 'english')
+        return message;
+    try {
+        const translation = await translate(message, { to: 'english' });
+        return translation.text || message;
+    } catch (error) {
+        console.error('Error translating message:', error);
+        return message;
+    }
+}