Crafting minds for Minecraft with LLMs and mineflayer!

FAQ | Discord Support | Blog Post | Paper Website | MineCollab

The open-source platform for crafting intelligent, collaborative agents in Minecraft using Large Language Models.

• Minecraft Java Edition (up to v1.21.4)
• node.js (at least v18)
• git
  One of these (optional, you can also use pollinations.ai without any key):

  • OpenAI API Key
  • Gemini API Key
  • Anthropic API Key
  • Replicate API Key
  • Hugging Face API Key
  • Groq API Key
  • Ollama Installed
  • Mistral API Key
  • Qwen API Key [Intl.] / [cn]
  • Doubao API Key [Intl.] / [cn]
  • Novita AI API Key

1. Make sure you have the requirements above.

2. Download this repository's latest release. Unzip it to your Downloads folder.

3. Rename keys.example.json to keys.json and fill in your API keys (you only need one). The desired model is set in andy.json or other profiles. For other models refer to the table below.

4. In terminal/command prompt, run npm install from the installed directory. (Note: If naudiodon fails to build and you don't need STT, you can usually proceed.)

5. Start a minecraft world and open it to LAN on localhost port 55916

6. Run node main.js from the installed directory

If you encounter issues, check the FAQ or find support on discord. If that fails, you can create an issue.

You can configure project details in settings.js. See file.

You can configure the agent's name, model, and prompts in their profile like andy.json with the model field. For comprehensive details, see Model Specifications.

If you use Ollama, to install the models used by default (generation and embedding), execute the following terminal command: ollama pull sweaterdog/andy-4 && ollama pull nomic-embed-text

Bot profiles are json files (such as andy.json) that define:

1. Bot backend LLMs to use for talking, coding, and embedding.
2. Prompts used to influence the bot's behavior.
3. Examples help the bot perform tasks.

LLM models can be specified simply as "model": "gpt-4o". However, you can use different models for chat, coding, and embeddings. You can pass a string or an object for these fields. A model object must specify an api, and optionally a model, url, and additional params.

"model": {
  "api": "openai",
  "model": "gpt-4.1",
  "url": "https://api.openai.com/v1/",
  "params": {
    "max_tokens": 1000,
    "temperature": 1
  }
},
"code_model": {
  "api": "openai",
  "model": "o4-mini",
  "url": "https://api.openai.com/v1/"
},
"vision_model": {
  "api": "openai",
  "model": "gpt-4.1",
  "url": "https://api.openai.com/v1/"
},
"embedding": {
  "api": "openai",
  "url": "https://api.openai.com/v1/",
  "model": "text-embedding-3-large"
},
"speak_model": {
  "api": "pollinations",
  "url": "https://text.pollinations.ai/openai",
  "model": "openai-audio",
  "voice": "echo"
}

model is used for chat, code_model is used for newAction coding, vision_model is used for image interpretation, and embedding is used to embed text for example selection. If code_model or vision_model is not specified, model will be used by default. Not all APIs support embeddings or vision.

All apis have default models and urls, so those fields are optional. The params field is optional and can be used to specify additional parameters for the model. It accepts any key-value pairs supported by the api. Is not supported for embedding models.

Embedding models are used to embed and efficiently select relevant examples for conversation and coding.

Supported Embedding APIs: openai, google, replicate, huggingface, novita, ollama, andy

If you try to use an unsupported model, then it will default to a simple word-overlap method. Expect reduced performance, recommend mixing APIs to ensure embedding support.

mindcraft-ce has support for custom plugins! For instructions, check out the plugin documentation.

To connect to online servers your bot will need an official Microsoft/Minecraft account. You can use your own personal one, but will need another account if you want to connect too and play with it. To connect, change these lines in settings.js:

"host": "111.222.333.444",
"port": 25565,
"auth": "microsoft",

// rest is same...

When using a Microsoft account for mindcraft, it will show a link and a code. Open the link in the browser, sign in with the Microsoft account you wish for the bot to use, and follow the on-screen instructions.

1. Clone the fork with the PR (e.g. mindcraft-fork), if you haven't already.
2. Add mindcraft-ce as a remote:

git remote add mindcraft-ce https://github.com/mindcraft-ce/mindcraft-ce.git

3. Push the branch to mindcraft-ce, replacing patch-x with your branch's name:

git push mindcraft-ce patch-x

4. On GitHub, go to mindcraft-ce, switch to patch-x, and create a PR to the desired branch in mindcraft-ce.

If you intend to allow_insecure_coding, it is a good idea to run the app in a docker container to reduce risks of running unknown code. This is strongly recommended before connecting to remote servers.

docker run -i -t --rm -v $(pwd):/app -w /app -p 3000-3003:3000-3003 node:latest node main.js

or simply

docker-compose up

When running in docker, if you want the bot to join your local minecraft server, you have to use a special host address host.docker.internal to call your localhost from inside your docker container. Put this into your settings.js:

"host": "host.docker.internal", // instead of "localhost", to join your local minecraft from inside the docker container

To connect to an unsupported minecraft version, you can try to use viaproxy

STT allows you to speak to the model if you have a microphone.

STT can be enabled in settings.js under the section that looks like this:

    "stt_transcription": true, // Change this to "true" to enable STT
    "stt_provider": "groq", // STT provider: "groq" or "pollinations"
    "stt_username": "SYSTEM",
    "stt_agent_name": ""

The Text to Speech engine will begin listening on the system default input device.

If for some reason STT does not work, install naudiodon by running the command: npm install naudiodon

STT Providers:

• Groq: You need a GroqCloud API key as Groq is used for Audio transcription
• Pollinations: Free STT service, no API key required. Uses the openai-audio model via the Pollinations API.

To use Groq STT, simply set "stt_provider": "groq" in your settings.js file. This provides an alternative to pollinations for speech-to-text transcription.

Mindcraft has the capabilities to collect data from you playing with the bots, which can be used to generate training data to fine-tune models such as Andy-4. To do this, enable logging inside of settings.js, then navigate to the logs folder.

Inside of the logs folder, and installing the dependecies, you will find a file named generate_usernames.py, you need to run this in order to convert your collected data into a usable dataset. This will generate a bunch of random names to replace the name of your bot, and your username. Both of which improve performance later on.

To run it, run python generate_usernames.py. The max amount of usernames will take up multiple Terabytes of data. If for some reason you want to do this, run it with the --make_all flag.

Next, you need to set up convert.py to include every username that interacted with the bot, as well as the bot's own username. This is done by adding / changing the usernames in the ORIGINAL_USERNAMES list.

After this, you are all set up for conversion! Since you might not want to convert all data at once, you must change the names of the .csv file*(s)* that you want to convert to Andy_pre1. If more than one file is wanted for conversion, change 1 to the next number, this value can be as high as you want.

To convert, run python convert.py, if you get a dependency error, ensure you are in a virtual python environment rather than a global one.

For setting up vision datasets, run convert.py with the flag of --vision, this will do the same thing as the rest of the conversions, but change the format to an image-friendly way. But it should be known that the formatted image data is not yet prepared for training, we are still working out how to have the data effectively be used by Unsloth.

The Andy API is a revolutionary distributed compute pool that allows users to share their AI resources and access models from around the world. By connecting to the Andy API network, you can:

• Contribute Resources: Share your local AI models (Ollama, LM Studio, etc.) or API quotas (OpenAI, Anthropic, etc.) with the community
• Access Diverse Models: Use models from other contributors without needing to host them locally
• Scale Dynamically: Automatically distribute workload across available compute resources

The Andy API Local Client is a modern web-based interface that makes it easy to connect any OpenAI-compatible endpoint to the distributed compute pool:

🔗 Repository: https://github.com/mindcraft-ce/Andy-API

Key Features:

• Universal Compatibility: Works with Ollama, OpenAI API, LM Studio, vLLM, and any OpenAI-compatible endpoint
• Web Dashboard: Real-time monitoring, model management, and performance analytics
• Easy Setup: Simple installation with automatic model discovery
• Resource Sharing: Contribute your compute power or API quotas to help the community

Quick Start:

# Clone the Andy API Local Client
git clone https://github.com/mindcraft-ce/Andy-API.git
cd Andy-API

# Install and run
pip install -r requirements.txt
python launch.py

# Open http://localhost:5000 in your browser

By running the Andy API Local Client alongside mindcraft-ce, you can contribute to the distributed AI ecosystem while using the best available models for your Minecraft agents!

Bot performance can be roughly evaluated with Tasks. Tasks automatically initialize bots with a goal to acquire specific items or construct predefined buildings, and remove the bot once the goal is achieved.

To run tasks, you need python, pip, and optionally conda. You can then install dependencies with pip install -r requirements.txt.

Tasks are defined in json files in the tasks folder, and can be run with: python tasks/run_task_file.py --task_path=tasks/example_tasks.json

For full evaluations, you will need to download and install the task suite. Full instructions.

By default, the program will use the profiles specified in settings.js. You can specify one or more agent profiles using the --profiles argument: node main.js --profiles ./profiles/andy.json ./profiles/jill.json

Some of the node modules that we depend on have bugs in them. To add a patch, change your local node module file and run npx patch-package [package-name]

@article{mindcraft2025,
  title = {Collaborating Action by Action: A Multi-agent LLM Framework for Embodied Reasoning},
  author = {White*, Isadora and Nottingham*, Kolby and Maniar, Ayush and Robinson, Max and Lillemark, Hansen and Maheshwari, Mehul and Qin, Lianhui and Ammanabrolu, Prithviraj},
  journal = {arXiv preprint arXiv:2504.17950},
  year = {2025},
  url = {https://arxiv.org/abs/2504.17950},
}