Add all the new component docs to the site and do a tidy up of docs (#7098)

* feat: add all the new component docs to the site

* fix(docs): relative links and markdown warnings

* feat(docs): How to contribute to the docs as a docs section

* fix(docs): missed docs page for developer setup

* fix(docs): re-add configurations options

* fix(docs): bad link to components fixed

* fix(docs): bad link to components fixed

* ref(docs): reorder some items to make more sense

* fix(docs): bad indentation and duplicate block

* fix(docs): warning about out of date markdown extension

* fix(docs): broken links fixed

* fix(docs): markdown formatter complaints

18 files changed, 159 insertions, 92 deletions

diff --git a/docs/content/AutoGPT/component agent/advanced.md b/docs/content/AutoGPT/components/advanced.md
index 090b811c2..2f4513c27 100644
--- a/docs/content/AutoGPT/component agent/advanced.md
+++ b/docs/content/AutoGPT/components/advanced.md
@@ -1,8 +1,11 @@
-### Other stuff
+# Advanced Components
+
+## General
+
 Debugging may be easier because we can inspect the exact components that were called and where the pipeline failed (current WIP pipeline):
 
-![](../imgs/modular-pipeline.png)
+![Modular Pipeline](../../imgs/modular-pipeline.png)
 
 Also that makes it possible to call component/pipeline/function again when failed and recover.
 
-If it's necessary to get a component in a random place, agent provides generic, type safe `get_component(type[T]) -> T | None`
\ No newline at end of file
+If it's necessary to get a component in a random place, agent provides generic, type safe `get_component(type[T]) -> T | None`
diff --git a/docs/content/AutoGPT/component agent/agents.md b/docs/content/AutoGPT/components/agents.md
index ca5c71b0c..4ab573243 100644
--- a/docs/content/AutoGPT/component agent/agents.md
+++ b/docs/content/AutoGPT/components/agents.md
@@ -34,4 +34,4 @@ class MyAgent(Agent):
         self.my_component = MyComponent()
 ```
 
-For more customization, you can override the `propose_action` and `execute` or even subclass `BaseAgent` directly. This way you can have full control over the agent's components and behavior. Have a look at the [implementation of Agent](./../../../../autogpts/autogpt/autogpt/agents/agent.py) for more details.
+For more customization, you can override the `propose_action` and `execute` or even subclass `BaseAgent` directly. This way you can have full control over the agent's components and behavior. Have a look at the [implementation of Agent](https://github.com/Significant-Gravitas/AutoGPT/tree/master/autogpts/autogpt/autogpt/agents/agent.py) for more details.
diff --git a/docs/content/AutoGPT/component agent/built-in-components.md b/docs/content/AutoGPT/components/built-in-components.md
index c46233049..c46233049 100644
--- a/docs/content/AutoGPT/component agent/built-in-components.md
+++ b/docs/content/AutoGPT/components/built-in-components.md
diff --git a/docs/content/AutoGPT/component agent/commands.md b/docs/content/AutoGPT/components/commands.md
index 68f3f3c42..68f3f3c42 100644
--- a/docs/content/AutoGPT/component agent/commands.md
+++ b/docs/content/AutoGPT/components/commands.md
diff --git a/docs/content/AutoGPT/component agent/components.md b/docs/content/AutoGPT/components/components.md
index 5575f0bcb..5575f0bcb 100644
--- a/docs/content/AutoGPT/component agent/components.md
+++ b/docs/content/AutoGPT/components/components.md
diff --git a/docs/content/AutoGPT/component agent/creating-components.md b/docs/content/AutoGPT/components/creating-components.md
index d06714beb..2eed5c683 100644
--- a/docs/content/AutoGPT/component agent/creating-components.md
+++ b/docs/content/AutoGPT/components/creating-components.md
@@ -94,7 +94,7 @@ To learn more about commands see [🛠️ Commands](./commands.md).
 After components provided all necessary data, the agent needs to build the final prompt that will be send to a llm.
 Currently, `PromptStrategy` (*not* a protocol) is responsible for building the final prompt.
 If you want to change the way the prompt is built, you need to create a new `PromptStrategy` class, and then call relavant methods in your agent class.
-You can have a look at the default strategy used by the AutoGPT Agent: [OneShotAgentPromptStrategy](../../../../autogpts/autogpt/autogpt/agents/prompt_strategies/one_shot.py), and how it's used in the [Agent](../../../../autogpts/autogpt/autogpt/agents/agent.py) (search for `self.prompt_strategy`).
+You can have a look at the default strategy used by the AutoGPT Agent: [OneShotAgentPromptStrategy](https://github.com/Significant-Gravitas/AutoGPT/tree/master/autogpts/autogpt/autogpt/agents/prompt_strategies/one_shot.py), and how it's used in the [Agent](https://github.com/Significant-Gravitas/AutoGPT/tree/master/autogpts/autogpt/autogpt/agents/agent.py) (search for `self.prompt_strategy`).
 
 ## Example `UserInteractionComponent`
 
@@ -102,53 +102,58 @@ Let's create a slighlty simplified version of the component that is used by the
 It gives an ability for the agent to ask user for input in the terminal.
 
 1. Create a class for the component that inherits from `CommandProvider`.
-```py
-class MyUserInteractionComponent(CommandProvider):
-    """Provides commands to interact with the user."""
-    pass
-```
+
+    ```py
+    class MyUserInteractionComponent(CommandProvider):
+        """Provides commands to interact with the user."""
+        pass
+    ```
 
 2. Implement command method that will ask user for input and return it.
-```py
-def ask_user(self, question: str) -> str:
-    """If you need more details or information regarding the given goals,
-    you can ask the user for input."""
-    print(f"\nQ: {question}")
-    resp = input("A:")
-    return f"The user's answer: '{resp}'"
-```
+
+    ```py
+    def ask_user(self, question: str) -> str:
+        """If you need more details or information regarding the given goals,
+        you can ask the user for input."""
+        print(f"\nQ: {question}")
+        resp = input("A:")
+        return f"The user's answer: '{resp}'"
+    ```
 
 3. The command needs to be decorated with `@command`.
-```py
-@command(
-    parameters={
-        "question": JSONSchema(
-            type=JSONSchema.Type.STRING,
-            description="The question or prompt to the user",
-            required=True,
-        )
-    },
-)
-def ask_user(self, question: str) -> str:
-    """If you need more details or information regarding the given goals,
-    you can ask the user for input."""
-    print(f"\nQ: {question}")
-    resp = input("A:")
-    return f"The user's answer: '{resp}'"
-```
+
+    ```py
+    @command(
+        parameters={
+            "question": JSONSchema(
+                type=JSONSchema.Type.STRING,
+                description="The question or prompt to the user",
+                required=True,
+            )
+        },
+    )
+    def ask_user(self, question: str) -> str:
+        """If you need more details or information regarding the given goals,
+        you can ask the user for input."""
+        print(f"\nQ: {question}")
+        resp = input("A:")
+        return f"The user's answer: '{resp}'"
+    ```
 
 4. We need to implement `CommandProvider`'s `get_commands` method to yield the command.
-```py
-def get_commands(self) -> Iterator[Command]:
-    yield self.ask_user
-```
+
+    ```py
+    def get_commands(self) -> Iterator[Command]:
+        yield self.ask_user
+    ```
 
 5. Since agent isn't always running in the terminal or interactive mode, we need to disable this component by setting `self._enabled` when it's not possible to ask for user input.
-```py
-def __init__(self, config: Config):
-    self.config = config
-    self._enabled = not config.noninteractive_mode
-```
+
+    ```py
+    def __init__(self, config: Config):
+        self.config = config
+        self._enabled = not config.noninteractive_mode
+    ```
 
 The final component should look like this:
 
@@ -228,7 +233,7 @@ class MyAgent(Agent):
 
 ## Learn more
 
-The best place to see more examples is to look at the built-in components in the [autogpt/components](../../../../autogpts/autogpt/autogpt/components/) and [autogpt/commands](../../../../autogpts/autogpt/autogpt/commands/) directories.
+The best place to see more examples is to look at the built-in components in the [autogpt/components](https://github.com/Significant-Gravitas/AutoGPT/tree/master/autogpts/autogpt/autogpt/components/) and [autogpt/commands](https://github.com/Significant-Gravitas/AutoGPT/tree/master/autogpts/autogpt/autogpt/commands/) directories.
 
 Guide on how to extend the built-in agent and build your own: [🤖 Agents](./agents.md)  
 Order of some components matters, see [🧩 Components](./components.md) to learn more about components and how they can be customized.  
diff --git a/docs/content/AutoGPT/component agent/introduction.md b/docs/content/AutoGPT/components/introduction.md
index 954b67770..954b67770 100644
--- a/docs/content/AutoGPT/component agent/introduction.md
+++ b/docs/content/AutoGPT/components/introduction.md
diff --git a/docs/content/AutoGPT/component agent/protocols.md b/docs/content/AutoGPT/components/protocols.md
index b3d0be8cf..b3d0be8cf 100644
--- a/docs/content/AutoGPT/component agent/protocols.md
+++ b/docs/content/AutoGPT/components/protocols.md
diff --git a/docs/content/AutoGPT/configuration/options.md b/docs/content/AutoGPT/configuration/options.md
index c84d70cfb..0ff2c29c9 100644
--- a/docs/content/AutoGPT/configuration/options.md
+++ b/docs/content/AutoGPT/configuration/options.md
@@ -11,7 +11,7 @@ Configuration is controlled through the `Config` object. You can set configurati
 - `BROWSE_CHUNK_MAX_LENGTH`: When browsing website, define the length of chunks to summarize. Default: 3000
 - `BROWSE_SPACY_LANGUAGE_MODEL`: [spaCy language model](https://spacy.io/usage/models) to use when creating chunks. Default: en_core_web_sm
 - `CHAT_MESSAGES_ENABLED`: Enable chat messages. Optional
-- `DISABLED_COMMANDS`: Commands to disable. Use comma separated names of commands. See the list of commands from built-in components [here](../component%20agent/components.md). Default: None
+- `DISABLED_COMMANDS`: Commands to disable. Use comma separated names of commands. See the list of commands from built-in components [here](../components/components.md). Default: None
 - `ELEVENLABS_API_KEY`: ElevenLabs API Key. Optional.
 - `ELEVENLABS_VOICE_ID`: ElevenLabs Voice ID. Optional.
 - `EMBEDDING_MODEL`: LLM Model to use for embedding tasks. Default: `text-embedding-3-small`
diff --git a/docs/content/AutoGPT/index.md b/docs/content/AutoGPT/index.md
index aad7cc37d..c53589baa 100644
--- a/docs/content/AutoGPT/index.md
+++ b/docs/content/AutoGPT/index.md
@@ -1,6 +1,6 @@
 # AutoGPT Agent
 
-[🔧 **Setup**](./setup)
+[🔧 **Setup**](setup/index.md)
  | 
 [💻 **User guide**](./usage.md)
  | 
diff --git a/docs/content/AutoGPT/setup/docker.md b/docs/content/AutoGPT/setup/docker.md
index af4794a25..e73048696 100644
--- a/docs/content/AutoGPT/setup/docker.md
+++ b/docs/content/AutoGPT/setup/docker.md
@@ -191,4 +191,4 @@ docker run -it --env-file=.env -v $PWD:/app autogpt
 docker run -it --env-file=.env -v $PWD:/app --rm autogpt --gpt3only --continuous
 ```
 
-[user guide]: /autogpt/usage/#command-line-interface
+[user guide]: ../usage.md/#command-line-interface
diff --git a/docs/content/AutoGPT/setup/index.md b/docs/content/AutoGPT/setup/index.md
index 5ef214b61..540c7f116 100644
--- a/docs/content/AutoGPT/setup/index.md
+++ b/docs/content/AutoGPT/setup/index.md
@@ -46,7 +46,7 @@ Get your OpenAI API key from:
     It's highly recommended that you keep track of your API costs on [the Usage page](https://platform.openai.com/account/usage).
     You can also set limits on how much you spend on [the Usage limits page](https://platform.openai.com/account/billing/limits).
 
-![For OpenAI API key to work, set up paid account at OpenAI API > Billing](/imgs/openai-api-key-billing-paid-account.png)
+![For OpenAI API key to work, set up paid account at OpenAI API > Billing](../../imgs/openai-api-key-billing-paid-account.png)
 
 
 ## Setting up AutoGPT
@@ -58,8 +58,8 @@ Since we don't ship AutoGPT as a desktop application, you'll need to download th
 
 ![Screenshot of the dialog to clone or download the repo](get-repo-dialog.png)
 
-* To get the latest bleeding edge version, use `master`.
-* If you're looking for more stability, check out the latest AutoGPT [release][releases].
+- To get the latest bleeding edge version, use `master`.
+- If you're looking for more stability, check out the latest AutoGPT [release][releases].
 
 [project]: https://github.com/Significant-Gravitas/AutoGPT
 [releases]: https://github.com/Significant-Gravitas/AutoGPT/releases
diff --git a/docs/content/AutoGPT/share-your-logs.md b/docs/content/AutoGPT/share-your-logs.md
index 3f4139ce1..279c6da92 100644
--- a/docs/content/AutoGPT/share-your-logs.md
+++ b/docs/content/AutoGPT/share-your-logs.md
@@ -1,9 +1,10 @@
-## Share your logs with us to help improve AutoGPT
+# Share your logs with us to help improve AutoGPT
 
 Do you notice weird behavior with your agent? Do you have an interesting use case? Do you have a bug you want to report?
 Follow the steps below to enable your logs and upload them. You can include these logs when making an issue report or discussing an issue with us.
 
-### Enable Debug Logs
+## Enable Debug Logs
+
 Activity, Error, and Debug logs are located in `./logs`
 
 To print out debug logs:
@@ -16,20 +17,19 @@ To print out debug logs:
 docker-compose run --rm auto-gpt --debug    # in Docker
 ```
 
-### Inspect and share logs
-You can inspect and share logs via [e2b](https://e2b.dev).
-![E2b logs dashboard](./imgs/e2b-dashboard.png)
-
+## Inspect and share logs
 
+You can inspect and share logs via [e2b](https://e2b.dev).
+![E2b logs dashboard](../imgs/e2b-dashboard.png)
 
 1. Go to [autogpt.e2b.dev](https://autogpt.e2b.dev) and sign in.
 2. You'll see logs from other members of the AutoGPT team that you can inspect.
 3. Or you upload your own logs. Click on the "Upload log folder" button and select the debug logs dir that you generated. Wait a 1-2 seconds and the page reloads.
 4. You can share logs via sharing the URL in your browser.
-![E2b log URL](./imgs/e2b-log-url.png)
-
+![E2b log URL](../imgs/e2b-log-url.png)
 
 ### Add tags to logs
+
 You can add custom tags to logs for other members of your team. This is useful if you want to indicate that the agent is for example having issues with challenges.
 
 E2b offers 3 types of severity:
@@ -41,12 +41,13 @@ E2b offers 3 types of severity:
 You can name your tag any way you want.
 
 #### How to add a tag
+
 1. Click on the "plus" button on the left from the logs folder name.
 
-![E2b tag button](./imgs/e2b-tag-button.png)
+    ![E2b tag button](../imgs/e2b-tag-button.png)
 
-2. Type the name of a new tag.
+1. Type the name of a new tag.
 
-3. Select the severity.
+1. Select the severity.
 
-![E2b new tag](./imgs/e2b-new-tag.png)
+    ![E2b new tag](../imgs/e2b-new-tag.png)
diff --git a/docs/content/AutoGPT/usage.md b/docs/content/AutoGPT/usage.md
index 6e7b81587..5a19dd9cb 100644
--- a/docs/content/AutoGPT/usage.md
+++ b/docs/content/AutoGPT/usage.md
@@ -227,5 +227,5 @@ For example, to disable python coding features, set it to the value below:
 DISABLED_COMMANDS=execute_python_code,execute_python_file
 ```
 
-[components]: ./component%20agent/components.md
-[commands]: ./component%20agent/built-in-components.md
\ No newline at end of file
+[components]: ./components/components.md
+[commands]: ./components/built-in-components.md
\ No newline at end of file
diff --git a/docs/content/challenges/list.md b/docs/content/challenges/list.md
index fedb1f8cd..e517cfe44 100644
--- a/docs/content/challenges/list.md
+++ b/docs/content/challenges/list.md
@@ -1,5 +1,5 @@
 # List of Challenges
 
-This page contains a curated list of challenges that AutoGPT currently faces. If you think you have a solution or idea to tackle any of these challenges, feel free to dive in and start working on them! New challenges can also be submitted by following the guidelines on the [Submit a Challenge](challenges/submit.md) page.
+This page contains a curated list of challenges that AutoGPT currently faces. If you think you have a solution or idea to tackle any of these challenges, feel free to dive in and start working on them! New challenges can also be submitted by following the guidelines on the [Submit a Challenge](./submit.md) page.
 
 Memory Challenges: [List of Challenges](memory/introduction.md)
diff --git a/docs/content/docs/index.md b/docs/content/docs/index.md
new file mode 100644
index 000000000..4cce72aac
--- /dev/null
+++ b/docs/content/docs/index.md
@@ -0,0 +1,48 @@
+# Contributing to the Docs
+
+We welcome contributions to our documentation! If you would like to contribute, please follow the steps below.
+
+## Setting up the Docs
+
+1. Clone the repository:
+
+    ```shell
+    git clone github.com/Significant-Gravitas/AutoGPT.git
+    ```
+
+1. Install the dependencies:
+
+    ```shell
+    python -m pip install -r docs/requirements.txt
+    ```
+
+    or
+
+    ```shell
+    python3 -m pip install -r docs/requirements.txt
+    ```
+
+1. Start iterating using mkdocs' live server:
+
+    ```shell
+    mkdocs serve
+    ```
+
+1. Open your browser and navigate to `http://127.0.0.1:8000`.
+
+1. The server will automatically reload the docs when you save your changes.
+
+## Adding a new page
+
+1. Create a new markdown file in the `docs/content` directory.
+1. Add the new page to the `nav` section in the `mkdocs.yml` file.
+1. Add the content to the new markdown file.
+1. Run `mkdocs serve` to see your changes.
+
+## Checking links
+
+To check for broken links in the documentation, run `mkdocs build` and look for warnings in the console output.
+
+## Submitting a Pull Request
+
+When you're ready to submit your changes, please create a pull request. We will review your changes and merge them if they are appropriate.
diff --git a/docs/content/forge/get-started.md b/docs/content/forge/get-started.md
index ca5af2b07..d50b8e5ff 100644
--- a/docs/content/forge/get-started.md
+++ b/docs/content/forge/get-started.md
@@ -1,18 +1,20 @@
 # AutoGPT Forge

 

-### 🛠️ Build your own Agent 🛠️

+## 🛠️ Build your own Agent 🛠️

 

 !!! warning

-    If you are trying to use AutoGPT this is not the tutorial for you! You need to use [this one](../../AutoGPT/setup/)

+    If you are trying to use AutoGPT this is not the tutorial for you! You need to use [this one](../AutoGPT/setup/index.md)

 

 Forge is a ready-to-go template for *your* agent application. All the boilerplate code is already handled, letting you channel all your creativity into the things that set *your* agent apart.

 

 ---

 

 ### 🛠️ **Why AutoGPT Forge?**

+

 - 💤 **No More Boilerplate!** - Don't let the mundane tasks stop you. Fork and build without the headache of starting from scratch!

 - 🧠 **Brain-centric Development!** - All the tools you need so you can spend 100% of your time on what matters - crafting the brain of your AI!

 - 🛠️ **Tooling ecosystem!** - We work with the best in class tools to bring you the best experience possible!

+

 ---

 

 ### 🚀 **Get Started!**

@@ -24,10 +26,8 @@ The getting started [tutorial series](https://aiedge.medium.com/autogpt-forge-e3
 3. [AutoGPT Forge: Interacting with your Agent](https://aiedge.medium.com/autogpt-forge-interacting-with-your-agent-1214561b06b)

 4. [AutoGPT Forge: Crafting Intelligent Agent Logic](https://medium.com/@aiedge/autogpt-forge-crafting-intelligent-agent-logic-bc5197b14cb4)

 

-

 Coming soon:

 

-

 5. Interacting with and Benchmarking your Agent

 6. Abilities

 7. The Planning Loop

diff --git a/docs/mkdocs.yml b/docs/mkdocs.yml
index 49605461b..324455827 100644
--- a/docs/mkdocs.yml
+++ b/docs/mkdocs.yml
@@ -6,32 +6,42 @@ nav:
   - Home: index.md
 
   - AutoGPT Agent:
-    - Introduction: AutoGPT/index.md
-    - Setup:
-        - Setting up AutoGPT: AutoGPT/setup/index.md
-        - Set up with Docker: AutoGPT/setup/docker.md
-    - Usage: AutoGPT/usage.md
-    - Plugins: AutoGPT/plugins.md
-    # - Configuration:
-    #     - Options: AutoGPT/configuration/options.md
-    #     - Search: AutoGPT/configuration/search.md
-    #     - Memory: AutoGPT/configuration/memory.md
-    #     - Voice: AutoGPT/configuration/voice.md
-    #     - Image Generation: AutoGPT/configuration/imagegen.md
-    - Help us improve AutoGPT:
-        - Share your debug logs with us: AutoGPT/share-your-logs.md
-        - Contribution guide: contributing.md
-        - Running tests: AutoGPT/testing.md
-        - Code of Conduct: code-of-conduct.md
+      - Introduction: AutoGPT/index.md
+      - Setup:
+          - Setting up AutoGPT: AutoGPT/setup/index.md
+          - Set up with Docker: AutoGPT/setup/docker.md
+          - For Developers: AutoGPT/setup/for-developers.md
+          - Configuration:
+              - Options: AutoGPT/configuration/options.md
+              - Search: AutoGPT/configuration/search.md
+              - Voice: AutoGPT/configuration/voice.md
+              - Image Generation: AutoGPT/configuration/imagegen.md
+      - Components:
+          - Introduction: AutoGPT/components/introduction.md
+          - Agents: AutoGPT/components/agents.md
+          - Components: AutoGPT/components/components.md
+          - Protocols: AutoGPT/components/protocols.md
+          - Commands: AutoGPT/components/commands.md
+          - Built in Components: AutoGPT/components/built-in-components.md
+          - Creating Components: AutoGPT/components/creating-components.md
+          - Advanced: AutoGPT/components/advanced.md
+      - Usage: AutoGPT/usage.md
+      - Help us improve AutoGPT:
+          - Share your debug logs with us: AutoGPT/share-your-logs.md
+          - Contribution guide: contributing.md
+          - Running tests: AutoGPT/testing.md
+          - Code of Conduct: code-of-conduct.md
 
   - Benchmark:
-    - Readme: https://github.com/Significant-Gravitas/AutoGPT/blob/master/benchmark/README.md
+      - Readme: https://github.com/Significant-Gravitas/AutoGPT/blob/master/benchmark/README.md
 
   - Forge:
-    - Introduction: forge/get-started.md
+      - Introduction: forge/get-started.md
 
   - Frontend:
-    - Readme: https://github.com/Significant-Gravitas/AutoGPT/blob/master/frontend/README.md
+      - Readme: https://github.com/Significant-Gravitas/AutoGPT/blob/master/frontend/README.md
+
+  - Docs: docs/index.md
 
   # - Challenges:
   #     - Introduction: challenges/introduction.md
@@ -98,8 +108,8 @@ markdown_extensions:
   - pymdownx.caret
   - pymdownx.details
   - pymdownx.emoji:
-      emoji_index: !!python/name:materialx.emoji.twemoji
-      emoji_generator: !!python/name:materialx.emoji.to_svg
+      emoji_index: !!python/name:material.extensions.emoji.twemoji
+      emoji_generator: !!python/name:material.extensions.emoji.to_svg
   - pymdownx.highlight
   - pymdownx.inlinehilite
   - pymdownx.keys