Merge pull request #82 from kolbytn/replicate-api

Replicate api

README.md

@@ -8,16 +8,21 @@ This project allows an AI model to write/execute code on your computer that may
 
 ## Requirements
 
-- [OpenAI API Subscription](https://openai.com/blog/openai-api), [Gemini API Subscription](https://aistudio.google.com/app/apikey), [Anthropic API Subscription](https://docs.anthropic.com/claude/docs/getting-access-to-claude), or [Ollama Installed](https://ollama.com/download)
+- [OpenAI API Subscription](https://openai.com/blog/openai-api), [Gemini API Subscription](https://aistudio.google.com/app/apikey), [Anthropic API Subscription](https://docs.anthropic.com/claude/docs/getting-access-to-claude), [Replicate API Subscription](https://replicate.com/) or [Ollama Installed](https://ollama.com/download)
 - [Minecraft Java Edition](https://www.minecraft.net/en-us/store/minecraft-java-bedrock-edition-pc)
 - [Node.js](https://nodejs.org/) (at least v14)
 
 ## Installation
 
-Add one of these environment variables:
+Add the environment variable for the model you want to use:
-  - `OPENAI_API_KEY` (and optionally `OPENAI_ORG_ID`)
+
-  - `GEMINI_API_KEY`
+| API | Env Variable | Example Model name | Docs |
-  - `ANTHROPIC_API_KEY` (and optionally `OPENAI_API_KEY` for embeddings. not necessary, but without embeddings performance will suffer)
+|------|------|------|------|
+| OpenAI | `OPENAI_API_KEY` | `gpt-3.5-turbo` | [docs](https://platform.openai.com/docs/models) | (optionally add `OPENAI_ORG_ID`)
+| Google | `GEMINI_API_KEY` | `gemini-pro` | [docs](https://ai.google.dev/gemini-api/docs/models/gemini) |
+| Anthropic | `ANTHROPIC_API_KEY` | `claude-3-haiku-20240307` | [docs](https://docs.anthropic.com/claude/docs/models-overview) |
+| Replicate | `REPLICATE_API_KEY` | `meta/meta-llama-3-70b-instruct` | [docs](https://replicate.com/collections/language-models) |
+| Ollama (local) | n/a | `llama3` | [docs](https://ollama.com/library) |
 
 ⭐[How do I add the API key as an environment variable?](https://phoenixnap.com/kb/windows-set-environment-variable)⭐
 
@@ -30,7 +35,7 @@ Run `npm install`
 
 Install the minecraft version specified in `settings.json`, currently supports up to 1.20.4
 
-## Running Locally
+### Running Locally
 
 Start a minecraft world and open it to LAN on localhost port `55916`
 
@@ -40,13 +45,27 @@ You can configure the agent's name, model, and prompts in their profile like `an
 
 You can configure project details in `settings.json`.
 
-## Bot Profiles
+### Online Servers
+To connect to online servers your bot will need an official Microsoft/Minecraft account. You can use your own personal one, but will need another account if you want to connect with it. Here is an example settings for this:
+```
+{
+    "minecraft_version": "1.20.4",
+    "host": "111.222.333.444",
+    "port": 55920,
+    "auth": "microsoft",
+    "allow_insecure_coding": false
+}
+```
+‼️Make sure your bot's name in the profile.json matches the account name! Otherwise the bot will spam talk to itself.
 
-Bot profiles are json files (such as `andy.json`) that define a bot's behavior in three ways:
+### Bot Profiles
+
+Bot profiles are json files (such as `andy.json`) that define:
 
 1. Bot backend LLMs to use for chat and embeddings.
 2. Prompts used to influence the bot's behavior.
-3. Examples retrieved and provided to the bot to help it better perform tasks.
+3. Examples help the bot perform tasks.
+
 
 ### Model Specifications
 
@@ -84,19 +103,6 @@ Thus, all the below specifications are equivalent to the above example:
 "embedding": "openai"
 ```
 
-## Online Servers
-To connect to online servers your bot will need an official Microsoft/Minecraft account. You can use your own personal one, but will need another account if you want to connect with it. Here is an example settings for this:
-```
-{
-    "minecraft_version": "1.20.4",
-    "host": "111.222.333.444",
-    "port": 55920,
-    "auth": "microsoft",
-    "allow_insecure_coding": false
-}
-```
-‼️Make sure your bot's name in the profile.json matches the account name! Otherwise the bot will spam talk to itself.
-
 ## Patches
 
 Some of the node modules that we depend on have bugs in them. To add a patch, change your local node module file and run `npx patch-package [package-name]`

andy.json

@@ -5,7 +5,7 @@
 
     "conversing": "You are a playful Minecraft bot named $NAME that can converse with players, see, move, mine, build, and interact with the world by using commands. Act human-like as if you were a typical Minecraft player, rather than an AI. Be very brief in your responses, don't apologize constantly, don't give instructions or make lists unless asked, and don't refuse requests. Don't pretend to act, use commands immediately when requested. Do NOT say this: 'Sure, I've stopped.', instead say this: 'Sure, I'll stop. !stop'. Do NOT say this: 'On my way! Give me a moment.', instead say this: 'On my way! !goToPlayer('playername', 3)'. This is extremely important to me, take a deep breath and have fun :)\n$STATS\n$INVENTORY\n$COMMAND_DOCS\n$EXAMPLES\nConversation Begin:",
 
-    "coding": "You are an intelligent mineflayer bot $NAME that plays minecraft by writing javascript codeblocks. Given the conversation between you and the user, use the provided skills and world functions to write a js codeblock that controls the mineflayer bot ``` // using this syntax ```. The code will be executed and you will recieve it's output. If you are satisfied with the response, respond without a codeblock in a conversational way. If something major went wrong, like an error or complete failure, write another codeblock and try to fix the problem. Minor mistakes are acceptable. Be maximally efficient, creative, and clear. Do not use commands !likeThis, only use codeblocks. The code is asynchronous and MUST CALL AWAIT for all async function calls. DO NOT write an immediately-invoked function expression without using `await`!! DO NOT WRITE LIKE THIS: ```(async () => {console.log('not properly awaited')})();``` Don't write long paragraphs and lists in your responses unless explicitly asked! Only summarize the code you write with a sentence or two when done. This is extremely important to me, take a deep breath and good luck! \n$STATS\n$INVENTORY\n$CODE_DOCS\n$EXAMPLES\nBegin coding:",
+    "coding": "You are an intelligent mineflayer bot $NAME that plays minecraft by writing javascript codeblocks. Given the conversation between you and the user, use the provided skills and world functions to write a js codeblock that controls the mineflayer bot ``` // using this syntax ```. The code will be executed and you will recieve it's output. If you are satisfied with the response, respond without a codeblock in a conversational way. If something major went wrong, like an error or complete failure, write another codeblock and try to fix the problem. Minor mistakes are acceptable. Be maximally efficient, creative, and clear. Do not use commands !likeThis, only use codeblocks. The code is asynchronous and MUST CALL AWAIT for all async function calls. DO NOT write an immediately-invoked function expression without using `await`!! DO NOT WRITE LIKE THIS: ```(async () => {console.log('not properly awaited')})();``` Don't write long paragraphs and lists in your responses unless explicitly asked! Only summarize the code you write with a sentence or two when done. This is extremely important to me, take a deep breath and good luck! \n$STATS\n$INVENTORY\n$CODE_DOCS\n$EXAMPLES\nConversation:",
 
     "saving_memory": "You are a minecraft bot named $NAME that has been talking and playing minecraft by using commands. Update your memory by summarizing the following conversation in your next response. Store information that will help you improve as a Minecraft bot. Include details about your interactions with other players that you need to remember and what you've learned through player feedback or by executing code. Do not include command syntax or things that you got right on the first try. Be extremely brief and use as few words as possible.\nOld Memory: '$MEMORY'\nRecent conversation: \n$TO_SUMMARIZE\nSummarize your old memory and recent conversation into a new memory, and respond only with the memory text: ",
 

package.json

@@ -12,6 +12,7 @@
         "mineflayer-pvp": "^1.3.2",
         "openai": "^4.4.0",
         "patch-package": "^8.0.0",
+        "replicate": "^0.29.4",
         "vec3": "^0.1.10",
         "yargs": "^17.7.2"
     },

src/agent/coder.js

@@ -91,6 +91,7 @@ export class Coder {
 
     async generateCodeLoop(agent_history) {
         let messages = agent_history.getHistory();
+        messages.push({role: 'system', content: 'Code generation started. Write code in codeblock in your response:'});
 
         let code_return = null;
         let failures = 0;
@@ -99,7 +100,7 @@ export class Coder {
             if (this.agent.bot.interrupt_code)
                 return interrupt_return;
             console.log(messages)
-            let res = await this.agent.prompter.promptCoding(messages);
+            let res = await this.agent.prompter.promptCoding(JSON.parse(JSON.stringify(messages)));
             if (this.agent.bot.interrupt_code)
                 return interrupt_return;
             let contains_code = res.indexOf('```') !== -1;

src/agent/history.js

@@ -22,7 +22,7 @@ export class History {
 
     async storeMemories(turns) {
         console.log("Storing memories...");
-        this.memory = await this.agent.prompter.promptMemSaving(this.memory, turns);
+        this.memory = await this.agent.prompter.promptMemSaving(this.getHistory(), turns);
         console.log("Memory updated to: ", this.memory);
     }
 

src/agent/prompter.js

@@ -8,6 +8,7 @@ import { getCommand } from './commands/index.js';
 import { Gemini } from '../models/gemini.js';
 import { GPT } from '../models/gpt.js';
 import { Claude } from '../models/claude.js';
+import { ReplicateAPI } from '../models/replicate.js';
 import { Local } from '../models/local.js';
 
 
@@ -28,6 +29,8 @@ export class Prompter {
                 chat.api = 'openai';
             else if (chat.model.includes('claude'))
                 chat.api = 'anthropic';
+            else if (chat.model.includes('meta/') || chat.model.includes('mistralai/') || chat.model.includes('replicate/'))
+                chat.api = 'replicate';
             else
                 chat.api = 'ollama';
         }
@@ -40,6 +43,8 @@ export class Prompter {
             this.chat_model = new GPT(chat.model, chat.url);
         else if (chat.api == 'anthropic')
             this.chat_model = new Claude(chat.model, chat.url);
+        else if (chat.api == 'replicate')
+            this.chat_model = new ReplicateAPI(chat.model, chat.url);
         else if (chat.api == 'ollama')
             this.chat_model = new Local(chat.model, chat.url);
         else
@@ -57,6 +62,8 @@ export class Prompter {
             this.embedding_model = new Gemini(embedding.model, embedding.url);
         else if (embedding.api == 'openai')
             this.embedding_model = new GPT(embedding.model, embedding.url);
+        else if (embedding.api == 'replicate') 
+            this.embedding_model = new ReplicateAPI(embedding.model, embedding.url);
         else if (embedding.api == 'ollama')
             this.embedding_model = new Local(embedding.model, embedding.url);
         else {

src/models/gemini.js

@@ -1,5 +1,5 @@
 import { GoogleGenerativeAI } from '@google/generative-ai';
-
+import { toSinglePrompt } from '../utils/text.js';
 
 export class Gemini {
     constructor(model_name, url) {
@@ -13,6 +13,7 @@ export class Gemini {
     }
 
     async sendRequest(turns, systemMessage) {
+        let model;
         if (this.url) {
             model = this.genAI.getGenerativeModel(
                 {model: this.model_name || "gemini-pro"},
@@ -24,23 +25,18 @@ export class Gemini {
             );
         }
 
-        const messages = [{'role': 'system', 'content': systemMessage}].concat(turns);
+        const stop_seq = '***';
-        let prompt = "";
+        const prompt = toSinglePrompt(turns, systemMessage, stop_seq, 'model');
-        let role = "";
-        messages.forEach((message) => {
-            role = message.role;
-            if (role === 'assistant') role = 'model';
-            prompt += `${role}: ${message.content}\n`;
-        });
-        if (role !== "model") // if the last message was from the user/system, add a prompt for the model. otherwise, pretend we are extending the model's own message
-            prompt += "model: ";
-        console.log(prompt)
         const result = await model.generateContent(prompt);
         const response = await result.response;
-        return response.text();
+        const text = response.text();
+        if (!text.includes(stop_seq)) return text;
+        const idx = text.indexOf(stop_seq);
+        return text.slice(0, idx);
     }
 
     async embed(text) {
+        let model;
         if (this.url) {
             model = this.genAI.getGenerativeModel(
                 {model: this.model_name || "embedding-001"},

src/models/replicate.js

@@ -0,0 +1,57 @@
+import Replicate from 'replicate';
+import { toSinglePrompt } from '../utils/text.js';
+
+// llama, mistral
+export class ReplicateAPI {
+	constructor(model_name, url) {
+		this.model_name = model_name;
+		this.url = url;
+
+		if (this.url) {
+			console.warn('Replicate API does not support custom URLs. Ignoring provided URL.');
+		}
+
+		if (!process.env.REPLICATE_API_KEY) {
+			throw new Error('Replicate API key missing! Make sure you set your REPLICATE_API_KEY environment variable.');
+		}
+
+		this.replicate = new Replicate({
+			auth: process.env.REPLICATE_API_KEY,
+		});
+	}
+
+	async sendRequest(turns, systemMessage) {
+		const stop_seq = '***';
+		const prompt = toSinglePrompt(turns, null, stop_seq);
+		let model_name = this.model_name || 'meta/meta-llama-3-70b-instruct';
+
+		const input = { prompt, system_prompt: systemMessage };
+		let res = null;
+		try {
+			console.log('Awaiting Replicate API response...');
+			let result = '';
+			for await (const event of this.replicate.stream(model_name, { input })) {
+				result += event;
+				if (result === '') break;
+				if (result.includes(stop_seq)) {
+					result = result.slice(0, result.indexOf(stop_seq));
+					break;
+				}
+			}
+			res = result;
+		} catch (err) {
+			console.log(err);
+			res = 'My brain disconnected, try again.';
+		}
+		console.log('Received.');
+		return res;
+	}
+
+	async embed(text) {
+		const output = await this.replicate.run(
+			this.model_name || "mark3labs/embeddings-gte-base:d619cff29338b9a37c3d06605042e1ff0594a8c3eff0175fd6967f5643fc4d47",
+			{ input: {text} }
+		);
+		return output.vectors;
+	}
+}

src/utils/text.js

@@ -12,3 +12,16 @@ export function stringifyTurns(turns) {
     }
     return res.trim();
 }
+
+export function toSinglePrompt(turns, system=null, stop_seq='***', model_nickname='assistant') {
+    let prompt = system ? `${system}${stop_seq}` : '';
+    let role = '';
+    turns.forEach((message) => {
+        role = message.role;
+        if (role === 'assistant') role = model_nickname;
+        prompt += `${role}: ${message.content}${stop_seq}`;
+    });
+    if (role !== model_nickname) // if the last message was from the user/system, add a prompt for the model. otherwise, pretend we are extending the model's own message
+        prompt += model_nickname + ": ";
+    return prompt;
+}