mirrors/mindcraft
1
0
Fork 0
mirror of https://github.com/kolbytn/mindcraft.git synced 2025-08-03 22:05:35 +02:00
Code Issues Projects Releases Packages Wiki Activity Actions

Merge branch 'main' into goals

Browse source
This commit is contained in:
MaxRobinsonTheGreat 2024-04-19 14:12:43 -05:00
commit 2e8f2545df
12 changed files with 228 additions and 44 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

18
README.md
View file

@ -8,7 +8,7 @@ This project allows an AI model to write/execute code on your computer that may
## Requirements
- [OpenAI API Subscription](https://openai.com/blog/openai-api) or [Gemini API Subscription](https://aistudio.google.com/app/apikey)
- [OpenAI API Subscription](https://openai.com/blog/openai-api), [Gemini API Subscription](https://aistudio.google.com/app/apikey), or [Anthropic API Subscription](https://docs.anthropic.com/claude/docs/getting-access-to-claude)
- [Minecraft Java Edition](https://www.minecraft.net/en-us/store/minecraft-java-bedrock-edition-pc)
- [Node.js](https://nodejs.org/) (at least v14)
@ -17,14 +17,18 @@ This project allows an AI model to write/execute code on your computer that may
Add one of these environment variables:
- `OPENAI_API_KEY` (and optionally `OPENAI_ORG_ID`)
- `GEMINI_API_KEY`
- `ANTHROPIC_API_KEY` (and optionally `OPENAI_API_KEY` for embeddings. not necessary, but without embeddings performance will suffer)
⭐[How do I add the API key as an environment variable?](https://phoenixnap.com/kb/windows-set-environment-variable)⭐
Clone/Download this repository
Run `npm install`
Install the minecraft version specified in `settings.json`, currently supports up to 1.20.2
Install the minecraft version specified in `settings.json`, currently supports up to 1.20.4
## Run
## Running Locally
Start a minecraft world and open it to LAN on localhost port `55916`
@ -32,17 +36,21 @@ Run `node main.js`
You can configure the agent's name, model, and prompts in their profile like `andy.json`.
You can configure project details in `settings.json`.
You can configure project details in `settings.json`. Here is an example settings for connecting to a non-local server:
## Online Servers
To connect to online servers your bot will need an official Microsoft/Minecraft account. You can use your own personal one, but will need another account if you want to connect with it. Here is an example settings for this:
```
{
"minecraft_version": "1.20.1",
"minecraft_version": "1.20.4",
"host": "111.222.333.444",
"port": 55920,
"auth": "microsoft",
"allow_insecure_coding": false
}
```
‼️Make sure your bot's name in the profile.json matches the account name! Otherwise the bot will spam talk to itself.
## Patches

10
andy.json
View file

@ -1,11 +1,11 @@
{
"name": "andy",
"model": "gpt-3.5-turbo-0125",
"model": "gpt-3.5-turbo",
"conversing": "You are a playful Minecraft bot named $NAME that can converse with players, see, move, mine, build, and interact with the world by using commands. Act human-like as if you were a typical Minecraft player, rather than an AI. Be very brief in your responses, don't apologize constantly, don't give instructions or make lists unless asked, and don't refuse requests. Don't pretend to act, use commands immediately when requested. Do NOT say this: 'Sure, I've stopped.', instead say this: 'Sure, I'll stop. !stop'. Do NOT say this: 'On my way! Give me a moment.', instead say this: 'On my way! !goToPlayer('playername', 3)'. Have fun :) \n$STATS\n$COMMAND_DOCS\n$EXAMPLES\nConversation Begin:",
"conversing": "You are a playful Minecraft bot named $NAME that can converse with players, see, move, mine, build, and interact with the world by using commands. Act human-like as if you were a typical Minecraft player, rather than an AI. Be very brief in your responses, don't apologize constantly, don't give instructions or make lists unless asked, and don't refuse requests. Don't pretend to act, use commands immediately when requested. Do NOT say this: 'Sure, I've stopped.', instead say this: 'Sure, I'll stop. !stop'. Do NOT say this: 'On my way! Give me a moment.', instead say this: 'On my way! !goToPlayer('playername', 3)'. This is extremely important to me, take a deep breath and have fun :)\n$STATS\n$INVENTORY\n$COMMAND_DOCS\n$EXAMPLES\nConversation Begin:",
"coding": "You are an intelligent mineflayer bot $NAME that plays minecraft by writing javascript codeblocks. Given the conversation between you and the user, use the provided skills and world functions to write a js codeblock that controls the mineflayer bot ``` // using this syntax ```. The code will be executed and you will recieve it's output. If you are satisfied with the response, respond without a codeblock in a conversational way. If something went wrong, write another codeblock and try to fix the problem. Be maximally efficient, creative, and clear. Do not use commands !likeThis, only use codeblocks.\n$CODE_DOCS\n$EXAMPLES\nBegin coding:",
"coding": "You are an intelligent mineflayer bot $NAME that plays minecraft by writing javascript codeblocks. Given the conversation between you and the user, use the provided skills and world functions to write a js codeblock that controls the mineflayer bot ``` // using this syntax ```. The code will be executed and you will recieve it's output. If you are satisfied with the response, respond without a codeblock in a conversational way. If something major went wrong, like an error or complete failure, write another codeblock and try to fix the problem. Minor mistakes are acceptable. Be maximally efficient, creative, and clear. Do not use commands !likeThis, only use codeblocks. Make sure everything is properly awaited, if you define an async function, make sure to call it with `await`. Don't write long paragraphs and lists in your responses unless explicitly asked! Only summarize the code you write with a sentence or two when done. This is extremely important to me, take a deep breath and good luck! \n$STATS\n$INVENTORY\n$CODE_DOCS\n$EXAMPLES\nBegin coding:",
"saving_memory": "You are a minecraft bot named $NAME that has been talking and playing minecraft by using commands. Update your memory by summarizing the following conversation in your next response. Store information that will help you improve as a Minecraft bot. Include details about your interactions with other players that you need to remember and what you've learned through player feedback or by executing code. Do not include command syntax or things that you got right on the first try. Be extremely brief and use as few words as possible.\nOld Memory: '$MEMORY'\nRecent conversation: \n$TO_SUMMARIZE\nSummarize your old memory and recent conversation into a new memory, and respond only with the memory text: ",
@ -70,7 +70,9 @@
[
{"role": "user", "content": "abc: stop"},
{"role": "assistant", "content": "Sure. !stop"}
{"role": "assistant", "content": "Sure. !stop"},
{"role": "system", "content": "Agent action stopped."},
{"role": "assistant", "content": "I've stopped! What next?"}
],
[

4
package.json
View file

@ -1,9 +1,10 @@
{
"type": "module",
"dependencies": {
"@anthropic-ai/sdk": "^0.17.1",
"@google/generative-ai": "^0.2.1",
"minecraft-data": "^3.46.2",
"mineflayer": "^4.14.0",
"mineflayer": "^4.20.0",
"mineflayer-armor-manager": "^2.0.1",
"mineflayer-auto-eat": "^3.3.6",
"mineflayer-collectblock": "^1.4.1",
@ -11,6 +12,7 @@
"mineflayer-pvp": "^1.3.2",
"openai": "^4.4.0",
"patch-package": "^8.0.0",
"vec3": "^0.1.10",
"yargs": "^17.7.2"
},
"scripts": {

2
settings.json
View file

@ -1,5 +1,5 @@
{
"minecraft_version": "1.20.1",
"minecraft_version": "1.20.4",
"host": "localhost",
"port": 55916,
"auth": "offline",

4
src/agent/agent.js
View file

@ -25,8 +25,8 @@ export class Agent {
initModes(this);
this.bot.on('login', async () => {
console.log(`${this.name} logged in.`);
this.bot.once('spawn', async () => {
console.log(`${this.name} spawned.`);
this.coder.clear();
const ignore_messages = [

14
src/agent/coder.js
View file

@ -94,14 +94,24 @@ export class Coder {
let code_return = null;
let failures = 0;
const interrupt_return = {success: true, message: null, interrupted: true, timedout: false};
for (let i=0; i<5; i++) {
if (this.agent.bot.interrupt_code)
return {success: true, message: null, interrupted: true, timedout: false};
return interrupt_return;
console.log(messages)
let res = await this.agent.prompter.promptCoding(messages);
console.log('Code generation response:', res)
if (this.agent.bot.interrupt_code)
return interrupt_return;
let contains_code = res.indexOf('```') !== -1;
if (!contains_code) {
if (res.indexOf('!newAction') !== -1) {
messages.push({
role: 'assistant',
content: res.substring(0, res.indexOf('!newAction'))
});
continue; // using newaction will continue the loop
}
if (code_return) {
agent_history.add('system', code_return.message);
agent_history.add(this.agent.name, res);

2
src/agent/commands/index.js
View file

@ -95,7 +95,7 @@ export async function executeCommand(agent, message) {
export function getCommandDocs() {
let docs = `\n*COMMAND DOCS\n You can use the following commands to perform actions and get information about the world.
Use the commands with the syntax: !commandName or !commandName("arg1", 1.2, ...) if the command takes arguments.\n
Do not use codeblocks. Only use one command in each response, trailing commands and comments will be ignored. Use these commands frequently in your responses!\n`;
Do not use codeblocks. Only use one command in each response, trailing commands and comments will be ignored.\n`;
for (let command of commandList) {
docs += command.name + ': ' + command.description + '\n';
if (command.params) {

108
src/agent/library/skills.js
View file

@ -18,8 +18,7 @@ async function autoLight(bot) {
if (has_torch) {
try {
log(bot, `Placing torch at ${bot.entity.position}.`);
await placeBlock(bot, 'torch', bot.entity.position.x, bot.entity.position.y, bot.entity.position.z);
return true;
return await placeBlock(bot, 'torch', bot.entity.position.x, bot.entity.position.y, bot.entity.position.z);
} catch (err) {return true;}
}
}
@ -491,13 +490,29 @@ export async function placeBlock(bot, blockType, x, y, z) {
* let position = world.getPosition(bot);
* await skills.placeBlock(bot, "oak_log", position.x + 1, position.y - 1, position.x);
**/
const target_dest = new Vec3(Math.floor(x), Math.floor(y), Math.floor(z));
const empty_blocks = ['air', 'water', 'lava', 'grass', 'tall_grass', 'snow', 'dead_bush', 'fern'];
const targetBlock = bot.blockAt(target_dest);
if (!empty_blocks.includes(targetBlock.name)) {
log(bot, `Cannot place block at ${targetBlock.position} because ${targetBlock.name} is in the way.`);
console.log('placing block...')
let block = bot.inventory.items().find(item => item.name === blockType);
if (!block) {
log(bot, `Don't have any ${blockType} to place.`);
return false;
}
const target_dest = new Vec3(Math.floor(x), Math.floor(y), Math.floor(z));
const targetBlock = bot.blockAt(target_dest);
if (targetBlock.name === blockType) {
log(bot, `${blockType} already at ${targetBlock.position}.`);
return false;
}
const empty_blocks = ['air', 'water', 'lava', 'grass', 'short_grass', 'tall_grass', 'snow', 'dead_bush', 'fern'];
if (!empty_blocks.includes(targetBlock.name)) {
log(bot, `${blockType} in the way at ${targetBlock.position}.`);
const removed = await breakBlockAt(bot, x, y, z);
if (!removed) {
log(bot, `Cannot place ${blockType} at ${targetBlock.position}: block in the way.`);
return false;
}
await new Promise(resolve => setTimeout(resolve, 200)); // wait for block to break
}
// get the buildoffblock and facevec based on whichever adjacent block is not empty
let buildOffBlock = null;
let faceVec = null;
@ -515,14 +530,9 @@ export async function placeBlock(bot, blockType, x, y, z) {
return false;
}
let block = bot.inventory.items().find(item => item.name === blockType);
if (!block) {
log(bot, `Don't have any ${blockType} to place.`);
return false;
}
const pos = bot.entity.position;
const pos_above = pos.plus(Vec3(0,1,0));
const dont_move_for = ['torch', 'redstone_torch', 'redstone', 'lever', 'button', 'rail', 'detector_rail', 'powered_rail', 'activator_rail', 'tripwire_hook', 'tripwire'];
const dont_move_for = ['torch', 'redstone_torch', 'redstone', 'lever', 'button', 'rail', 'detector_rail', 'powered_rail', 'activator_rail', 'tripwire_hook', 'tripwire', 'water_bucket'];
if (!dont_move_for.includes(blockType) && (pos.distanceTo(targetBlock.position) < 1 || pos_above.distanceTo(targetBlock.position) < 1)) {
// too close
let goal = new pf.goals.GoalNear(targetBlock.position.x, targetBlock.position.y, targetBlock.position.z, 2);
@ -533,7 +543,8 @@ export async function placeBlock(bot, blockType, x, y, z) {
if (bot.entity.position.distanceTo(targetBlock.position) > 4.5) {
// too far
let pos = targetBlock.position;
bot.pathfinder.setMovements(new pf.Movements(bot));
let movements = new pf.Movements(bot);
bot.pathfinder.setMovements(movements);
await bot.pathfinder.goto(new pf.goals.GoalNear(pos.x, pos.y, pos.z, 4));
}
@ -695,8 +706,9 @@ export async function goToPlayer(bot, username, distance=3) {
log(bot, `Could not find ${username}.`);
return false;
}
bot.pathfinder.setMovements(new pf.Movements(bot));
const move = new pf.Movements(bot);
bot.pathfinder.setMovements(move);
await bot.pathfinder.goto(new pf.goals.GoalFollow(player, distance), true);
log(bot, `You have reached ${username}.`);
@ -716,7 +728,8 @@ export async function followPlayer(bot, username, distance=4) {
if (!player)
return false;
bot.pathfinder.setMovements(new pf.Movements(bot));
const move = new pf.Movements(bot);
bot.pathfinder.setMovements(move);
bot.pathfinder.setGoal(new pf.goals.GoalFollow(player, distance), true);
log(bot, `You are now actively following player ${username}.`);
@ -838,3 +851,64 @@ export async function goToBed(bot) {
log(bot, `You have woken up.`);
return true;
}
export async function tillAndSow(bot, x, y, z, seedType=null) {
/**
* Till the ground at the given position and plant the given seed type.
* @param {MinecraftBot} bot, reference to the minecraft bot.
* @param {number} x, the x coordinate to till.
* @param {number} y, the y coordinate to till.
* @param {number} z, the z coordinate to till.
* @param {string} plantType, the type of plant to plant. Defaults to none, which will only till the ground.
* @returns {Promise<boolean>} true if the ground was tilled, false otherwise.
* @example
* let position = world.getPosition(bot);
* await skills.till(bot, position.x, position.y - 1, position.x);
**/
console.log(x, y, z)
x = Math.round(x);
y = Math.round(y);
z = Math.round(z);
let block = bot.blockAt(new Vec3(x, y, z));
console.log(x, y, z)
if (block.name !== 'grass_block' && block.name !== 'dirt' && block.name !== 'farmland') {
log(bot, `Cannot till ${block.name}, must be grass_block or dirt.`);
return false;
}
let above = bot.blockAt(new Vec3(x, y+1, z));
if (above.name !== 'air') {
log(bot, `Cannot till, there is ${above.name} above the block.`);
return false;
}
// if distance is too far, move to the block
if (bot.entity.position.distanceTo(block.position) > 4.5) {
let pos = block.position;
bot.pathfinder.setMovements(new pf.Movements(bot));
await bot.pathfinder.goto(new pf.goals.GoalNear(pos.x, pos.y, pos.z, 4));
}
if (block.name !== 'farmland') {
let hoe = bot.inventory.items().find(item => item.name.includes('hoe'));
if (!hoe) {
log(bot, `Cannot till, no hoes.`);
return false;
}
await bot.equip(hoe, 'hand');
await bot.activateBlock(block);
log(bot, `Tilled block x:${x.toFixed(1)}, y:${y.toFixed(1)}, z:${z.toFixed(1)}.`);
}
if (seedType) {
if (seedType.endsWith('seed') && !seedType.endsWith('seeds'))
seedType += 's'; // fixes common mistake
let seeds = bot.inventory.items().find(item => item.name === seedType);
if (!seeds) {
log(bot, `No ${seedType} to plant.`);
return false;
}
await bot.equip(seeds, 'hand');
await bot.placeBlock(block, new Vec3(0, -1, 0));
log(bot, `Planted ${seedType} at x:${x.toFixed(1)}, y:${y.toFixed(1)}, z:${z.toFixed(1)}.`);
}
return true;
}

14
src/agent/prompter.js
View file

@ -1,12 +1,14 @@
import { readFileSync, mkdirSync, writeFileSync} from 'fs';
import { Gemini } from '../models/gemini.js';
import { GPT } from '../models/gpt.js';
import { Examples } from '../utils/examples.js';
import { getCommandDocs } from './commands/index.js';
import { getSkillDocs } from './library/index.js';
import { stringifyTurns } from '../utils/text.js';
import { getCommand } from './commands/index.js';
import { Gemini } from '../models/gemini.js';
import { GPT } from '../models/gpt.js';
import { Claude } from '../models/claude.js';
export class Prompter {
constructor(agent, fp) {
@ -26,6 +28,8 @@ export class Prompter {
this.model = new Gemini(model_name);
else if (model_name.includes('gpt'))
this.model = new GPT(model_name);
else if (model_name.includes('claude'))
this.model = new Claude(model_name);
else
throw new Error('Unknown model ' + model_name);
}
@ -50,6 +54,10 @@ export class Prompter {
let stats = await getCommand('!stats').perform(this.agent);
prompt = prompt.replaceAll('$STATS', stats);
}
if (prompt.includes('$INVENTORY')) {
let inventory = await getCommand('!inventory').perform(this.agent);
prompt = prompt.replaceAll('$INVENTORY', inventory);
}
if (prompt.includes('$COMMAND_DOCS'))
prompt = prompt.replaceAll('$COMMAND_DOCS', getCommandDocs());
if (prompt.includes('$CODE_DOCS'))
@ -64,7 +72,7 @@ export class Prompter {
// check if there are any remaining placeholders with syntax $<word>
let remaining = prompt.match(/\$[A-Z_]+/g);
if (remaining !== null) {
console.warn('Unknown prompt placeholders:', remaining);
console.warn('Unknown prompt placeholders:', remaining.join(', '));
}
return prompt;
}

81
src/models/claude.js Normal file
View file

@ -0,0 +1,81 @@
import Anthropic from '@anthropic-ai/sdk';
import { GPT } from './gpt.js';
export class Claude {
constructor(model_name) {
this.model_name = model_name;
if (!process.env.ANTHROPIC_API_KEY) {
throw new Error('Anthropic API key missing! Make sure you set your ANTHROPIC_API_KEY environment variable.');
}
this.anthropic = new Anthropic({
apiKey: process.env["ANTHROPIC_API_KEY"]
});
this.gpt = undefined;
try {
this.gpt = new GPT(); // use for embeddings, ignore model
} catch (err) {
console.warn('Claude uses the OpenAI API for embeddings, but no OPENAI_API_KEY env variable was found. Claude will still work, but performance will suffer.');
}
}
async sendRequest(turns, systemMessage) {
let prev_role = null;
let messages = [];
let filler = {role: 'user', content: '_'};
for (let msg of turns) {
if (msg.role === 'system') {
msg.role = 'user';
msg.content = 'SYSTEM: ' + msg.content;
}
if (msg.role === prev_role && msg.role === 'assistant') {
// insert empty user message to separate assistant messages
messages.push(filler);
messages.push(msg);
}
else if (msg.role === prev_role) {
// combine new message with previous message instead of adding a new one
messages[messages.length-1].content += '\n' + msg.content;
}
else {
messages.push(msg);
}
prev_role = msg.role;
}
if (messages.length === 0) {
messages.push(filler);
}
let res = null;
try {
console.log('Awaiting anthropic api response...')
console.log('Messages:', messages);
const resp = await this.anthropic.messages.create({
model: this.model_name,
system: systemMessage,
max_tokens: 2048,
messages: messages,
});
console.log('Received.')
res = resp.content[0].text;
}
catch (err) {
console.log(err);
res = 'My brain disconnected, try again.';
}
return res;
}
async embed(text) {
if (this.gpt) {
return await this.gpt.embed(text);
}
// if no gpt, just return random embedding
return Array(1).fill().map(() => Math.random());
}
}

12
src/models/gemini.js
View file

@ -3,12 +3,13 @@ import { GoogleGenerativeAI } from '@google/generative-ai';
export class Gemini {
constructor(model_name) {
if (!process.env.GEMINI_API_KEY) {
console.error('Gemini API key missing! Make sure you set your GEMINI_API_KEY environment variable.');
process.exit(1);
throw new Error('Gemini API key missing! Make sure you set your GEMINI_API_KEY environment variable.');
}
this.genAI = new GoogleGenerativeAI(process.env.GEMINI_API_KEY);
this.model = this.genAI.getGenerativeModel({ model: model_name });
this.llmModel = this.genAI.getGenerativeModel({ model: model_name });
this.embedModel = this.genAI.getGenerativeModel({ model: "embedding-001"});
}
async sendRequest(turns, systemMessage) {
@ -23,14 +24,13 @@ export class Gemini {
if (role !== "model") // if the last message was from the user/system, add a prompt for the model. otherwise, pretend we are extending the model's own message
prompt += "model: ";
console.log(prompt)
const result = await this.model.generateContent(prompt);
const result = await this.llmModel.generateContent(prompt);
const response = await result.response;
return response.text();
}
async embed(text) {
const model = this.genAI.getGenerativeModel({ model: "embedding-001"});
const result = await model.embedContent(text);
const result = await this.embedModel.embedContent(text);
return result.embedding;
}
}

3
src/models/gpt.js
View file

@ -16,8 +16,7 @@ export class GPT {
};
}
else {
console.error('OpenAI API key missing! Make sure you set OPENAI_API_KEY and OPENAI_ORG_ID (optional) environment variables.');
process.exit(1);
throw new Error('OpenAI API key missing! Make sure you set your OPENAI_API_KEY environment variable.');
}
this.openai = new OpenAIApi(openAiConfig);