feat(agent): Component-based Agents (#7054)

This incremental re-architecture unifies Agent code and plugins, so everything is component-based.

## Breaking changes

- Removed command categories and `DISABLED_COMMAND_CATEGORIES` environment variable. Use `DISABLED_COMMANDS` environment variable to disable individual commands.
- Changed `command` decorator; old-style commands are no longer supported. Implement `CommandProvider` on components instead.
- Removed `CommandRegistry`, now all commands are provided by components implementing `CommandProvider`.
- Removed `prompt_config` from `AgentSettings`.
- Removed plugin support: old plugins will no longer be loaded and executed.
- Removed `PromptScratchpad`, it was used by plugins and is no longer needed.
- Changed `ThoughtProcessOutput` from tuple to pydantic `BaseModel`.

## Other changes

- Created `AgentComponent`, protocols and logic to execute them.
- `BaseAgent` and `Agent` is now composed of components.
- Moved some logic from `BaseAgent` to `Agent`.
- Moved agent features and commands to components.
- Removed check if the same operation is about to be executed twice in a row.
- Removed file logging from `FileManagerComponent` (formerly `AgentFileManagerMixin`)
- Updated tests
- Added docs

See [Introduction](https://github.com/kcze/AutoGPT/blob/kpczerwinski/open-440-modular-agents/docs/content/AutoGPT/component%20agent/introduction.md) for more information.

12 files changed, 818 insertions, 28 deletions

diff --git a/docs/content/AutoGPT/component agent/advanced.md b/docs/content/AutoGPT/component agent/advanced.md
new file mode 100644
index 000000000..090b811c2
--- /dev/null
+++ b/docs/content/AutoGPT/component agent/advanced.md
@@ -0,0 +1,8 @@
+### Other stuff
+Debugging may be easier because we can inspect the exact components that were called and where the pipeline failed (current WIP pipeline):
+
+![](../imgs/modular-pipeline.png)
+
+Also that makes it possible to call component/pipeline/function again when failed and recover.
+
+If it's necessary to get a component in a random place, agent provides generic, type safe `get_component(type[T]) -> T | None`
\ No newline at end of file
diff --git a/docs/content/AutoGPT/component agent/agents.md b/docs/content/AutoGPT/component agent/agents.md
new file mode 100644
index 000000000..ca5c71b0c
--- /dev/null
+++ b/docs/content/AutoGPT/component agent/agents.md
@@ -0,0 +1,37 @@
+# 🤖 Agents
+
+Agent is composed of [🧩 Components](./components.md) and responsible for executing pipelines and some additional logic. The base class for all agents is `BaseAgent`, it has the necessary logic to collect components and execute protocols.
+
+## Important methods
+
+`BaseAgent` provides two abstract methods needed for any agent to work properly:
+1. `propose_action`: This method is responsible for proposing an action based on the current state of the agent, it returns `ThoughtProcessOutput`.
+2. `execute`: This method is responsible for executing the proposed action, returns `ActionResult`.
+
+## AutoGPT Agent
+
+`Agent` is the main agent provided by AutoGPT. It's a subclass of `BaseAgent`. It has all the [Built-in Components](./built-in-components.md). `Agent` implements the essential abstract methods from `BaseAgent`: `propose_action` and `execute`.
+
+## Building your own Agent
+
+The easiest way to build your own agent is to extend the `Agent` class and add additional components. By doing this you can reuse the existing components and the default logic for executing [⚙️ Protocols](./protocols.md).
+
+```py
+class MyComponent(AgentComponent):
+    pass
+
+class MyAgent(Agent):
+    def __init__(
+        self,
+        settings: AgentSettings,
+        llm_provider: ChatModelProvider,
+        file_storage: FileStorage,
+        legacy_config: Config,
+    ):
+        # Call the parent constructor to bring in the default components
+        super().__init__(settings, llm_provider, file_storage, legacy_config)
+        # Add your custom component
+        self.my_component = MyComponent()
+```
+
+For more customization, you can override the `propose_action` and `execute` or even subclass `BaseAgent` directly. This way you can have full control over the agent's components and behavior. Have a look at the [implementation of Agent](./../../../../autogpts/autogpt/autogpt/agents/agent.py) for more details.
diff --git a/docs/content/AutoGPT/component agent/built-in-components.md b/docs/content/AutoGPT/component agent/built-in-components.md
new file mode 100644
index 000000000..c46233049
--- /dev/null
+++ b/docs/content/AutoGPT/component agent/built-in-components.md
@@ -0,0 +1,115 @@
+# Built-in Components
+
+This page lists all [🧩 Components](./components.md) and [⚙️ Protocols](./protocols.md) they implement that are natively provided. They are used by the AutoGPT agent.
+
+## `SystemComponent`
+
+Essential component to allow an agent to finish.
+
+**DirectiveProvider**
+- Constraints about API budget
+  
+**MessageProvider**
+- Current time and date
+- Remaining API budget and warnings if budget is low
+
+**CommandProvider**
+- `finish` used when task is completed
+
+## `UserInteractionComponent`
+
+Adds ability to interact with user in CLI.
+
+**CommandProvider**
+- `ask_user` used to ask user for input
+
+## `FileManagerComponent`
+
+Adds ability to read and write persistent files to local storage, Google Cloud Storage or Amazon's S3.
+Necessary for saving and loading agent's state (preserving session).
+
+**DirectiveProvider**
+- Resource information that it's possible to read and write files
+
+**CommandProvider**
+- `read_file` used to read file
+- `write_file` used to write file
+- `list_folder` lists all files in a folder 
+
+## `CodeExecutorComponent`
+
+Lets the agent execute non-interactive Shell commands and Python code. Python execution works only if Docker is available.
+
+**CommandProvider**
+- `execute_shell` execute shell command
+- `execute_shell_popen` execute shell command with popen
+- `execute_python_code` execute Python code
+- `execute_python_file` execute Python file
+
+## `EventHistoryComponent`
+
+Keeps track of agent's actions and their outcomes. Provides their summary to the prompt.
+
+**MessageProvider**
+- Agent's progress summary
+
+**AfterParse**
+- Register agent's action
+
+**ExecutionFailuer**
+- Rewinds the agent's action, so it isn't saved
+
+**AfterExecute**
+- Saves the agent's action result in the history
+
+## `GitOperationsComponent`
+
+**CommandProvider**
+- `clone_repository` used to clone a git repository
+
+## `ImageGeneratorComponent`
+
+Adds ability to generate images using various providers, see [Image Generation configuration](./../configuration/imagegen.md) to learn more.
+
+**CommandProvider**
+- `generate_image` used to generate an image given a prompt
+
+## `WebSearchComponent`
+
+Allows agent to search the web.
+
+**DirectiveProvider**
+- Resource information that it's possible to search the web
+
+**CommandProvider**
+- `search_web` used to search the web using DuckDuckGo
+- `google` used to search the web using Google, requires API key
+
+## `WebSeleniumComponent`
+
+Allows agent to read websites using Selenium.
+
+**DirectiveProvider**
+- Resource information that it's possible to read websites
+
+**CommandProvider**
+- `read_website` used to read a specific url and look for specific topics or answer a question
+
+## `ContextComponent`
+
+Adds ability to keep up-to-date file and folder content in the prompt.
+
+**MessageProvider**
+- Content of elements in the context
+
+**CommandProvider**
+- `open_file` used to open a file into context
+- `open_folder` used to open a folder into context
+- `close_context_item` remove an item from the context
+
+## `WatchdogComponent`
+
+Watches if agent is looping and switches to smart mode if necessary.
+
+**AfterParse**
+- Investigates what happened and switches to smart mode if necessary
\ No newline at end of file
diff --git a/docs/content/AutoGPT/component agent/commands.md b/docs/content/AutoGPT/component agent/commands.md
new file mode 100644
index 000000000..68f3f3c42
--- /dev/null
+++ b/docs/content/AutoGPT/component agent/commands.md
@@ -0,0 +1,102 @@
+# 🛠️ Commands
+
+Commands are a way for the agent to do anything; e.g. interact with the user or APIs and use tools. They are provided by components that implement the `CommandProvider` [⚙️ Protocol](./protocols.md). Commands are functions that can be called by the agent, they can have parameters and return values that will be seen by the agent.
+
+```py
+class CommandProvider(Protocol):
+    def get_commands(self) -> Iterator[Command]:
+        ...
+```
+
+## `command` decorator
+
+The easiest and recommended way to provide a command is to use `command` decorator on a component method and then just yield it in `get_commands` as part of your provider. Each command needs a name, description and a parameter schema - `JSONSchema`. By default method name is used as a command name, and first part of docstring for the description (before first double newline) and schema can be provided in the decorator.
+
+### Example usage of `command` decorator
+
+```py
+# Assuming this is inside some component class
+@command(
+    parameters={
+        "a": JSONSchema(
+            type=JSONSchema.Type.INTEGER,
+            description="The first number",
+            required=True,
+        ),
+        "b": JSONSchema(
+            type=JSONSchema.Type.INTEGER,
+            description="The second number",
+            required=True,
+        )})
+def multiply(self, a: int, b: int) -> str:
+    """
+    Multiplies two numbers.
+    
+    Args:
+        a: First number
+        b: Second number
+
+    Returns:
+        Result of multiplication
+    """
+    return str(a * b)
+```
+
+The agent will be able to call this command, named `multiply` with two arguments and will receive the result. The command description will be: `Multiplies two numbers.`
+
+We can provide `names` and `description` in the decorator, the above command is equivalent to:
+
+```py
+@command(
+    names=["multiply"],
+    description="Multiplies two numbers.",
+    parameters={
+        "a": JSONSchema(
+            type=JSONSchema.Type.INTEGER,
+            description="The first number",
+            required=True,
+        ),
+        "b": JSONSchema(
+            type=JSONSchema.Type.INTEGER,
+            description="The second number",
+            required=True,
+        )})
+    def multiply_command(self, a: int, b: int) -> str:
+        return str(a * b)
+```
+
+To provide the `multiply` command to the agent, we need to yield it in `get_commands`:
+
+```py
+def get_commands(self) -> Iterator[Command]:
+    yield self.multiply
+```
+
+## Creating `Command` directly
+
+If you don't want to use the decorator, you can create a `Command` object directly.
+
+```py
+
+def multiply(self, a: int, b: int) -> str:
+        return str(a * b)
+
+def get_commands(self) -> Iterator[Command]:
+    yield Command(
+        names=["multiply"],
+        description="Multiplies two numbers.",
+        method=self.multiply,
+        parameters=[
+            CommandParameter(name="a", spec=JSONSchema(
+                type=JSONSchema.Type.INTEGER,
+                description="The first number",
+                required=True,
+            )),
+            CommandParameter(name="b", spec=JSONSchema(
+                type=JSONSchema.Type.INTEGER,
+                description="The second number",
+                required=True,
+            )),
+        ],
+    )
+```
\ No newline at end of file
diff --git a/docs/content/AutoGPT/component agent/components.md b/docs/content/AutoGPT/component agent/components.md
new file mode 100644
index 000000000..5575f0bcb
--- /dev/null
+++ b/docs/content/AutoGPT/component agent/components.md
@@ -0,0 +1,128 @@
+# 🧩 Components
+
+Components are the building blocks of [🤖 Agents](./agents.md). They are classes inheriting `AgentComponent` or implementing one or more [⚙️ Protocols](./protocols.md) that give agent additional abilities or processing. 
+
+Components can be used to implement various functionalities like providing messages to the prompt, executing code, or interacting with external services.
+They can be enabled or disabled, ordered, and can rely on each other.
+
+Components assigned in the agent's `__init__` via `self` are automatically detected upon the agent's instantiation.
+For example inside `__init__`: `self.my_component = MyComponent()`.
+You can use any valid Python variable name, what matters for the component to be detected is its type (`AgentComponent` or any protocol inheriting from it).
+
+Visit [Built-in Components](./built-in-components.md) to see what components are available out of the box.
+
+```py
+from autogpt.agents import Agent
+from autogpt.agents.components import AgentComponent
+
+class HelloComponent(AgentComponent):
+    pass
+
+class SomeComponent(AgentComponent):
+    def __init__(self, hello_component: HelloComponent):
+        self.hello_component = hello_component
+
+class MyAgent(Agent):
+    def __init__(self):
+        # These components will be automatically discovered and used
+        self.hello_component = HelloComponent()
+        # We pass HelloComponent to SomeComponent
+        self.some_component = SomeComponent(self.hello_component)
+```
+
+## Ordering components
+
+The execution order of components is important because the latter ones may depend on the results of the former ones.
+
+### Implicit order
+
+Components can be ordered implicitly by the agent; each component can set `run_after` list to specify which components should run before it. This is useful when components rely on each other or need to be executed in a specific order. Otherwise, the order of components is alphabetical.
+
+```py
+# This component will run after HelloComponent
+class CalculatorComponent(AgentComponent):
+    run_after = [HelloComponent]
+```
+
+### Explicit order
+
+Sometimes it may be easier to order components explicitly by setting `self.components` list in the agent's `__init__` method. This way you can also ensure there's no circular dependencies and `run_after` is ignored.
+
+!!! warning
+    Be sure to include all components - by setting `self.components` list, you're overriding the default behavior of discovering components automatically. Since it's usually not intended agent will inform you in the terminal if some components were skipped.
+
+```py
+class MyAgent(Agent):
+    def __init__(self):
+        self.hello_component = HelloComponent()
+        self.calculator_component = CalculatorComponent(self.hello_component)
+        # Explicitly set components list
+        self.components = [self.hello_component, self.calculator_component]
+```
+
+## Disabling components
+
+You can control which components are enabled by setting their `_enabled` attribute.
+Either provide a `bool` value or a `Callable[[], bool]`, will be checked each time
+the component is about to be executed. This way you can dynamically enable or disable
+components based on some conditions.
+You can also provide a reason for disabling the component by setting `_disabled_reason`.
+The reason will be visible in the debug information.
+
+```py
+class DisabledComponent(MessageProvider):
+    def __init__(self):
+        # Disable this component
+        self._enabled = False
+        self._disabled_reason = "This component is disabled because of reasons."
+
+        # Or disable based on some condition, either statically...:
+        self._enabled = self.some_property is not None
+        # ... or dynamically:
+        self._enabled = lambda: self.some_property is not None
+
+    # This method will never be called
+    def get_messages(self) -> Iterator[ChatMessage]:
+        yield ChatMessage.user("This message won't be seen!")
+
+    def some_condition(self) -> bool:
+        return False
+```
+
+If you don't want the component at all, you can just remove it from the agent's `__init__` method. If you want to remove components you inherit from the parent class you can set the relevant attribute to `None`:
+
+!!! Warning
+    Be careful when removing components that are required by other components. This may lead to errors and unexpected behavior.
+
+```py
+class MyAgent(Agent):
+    def __init__(self):
+        super().__init__(...)
+        # Disable WatchdogComponent that is in the parent class
+        self.watchdog = None
+
+```
+
+## Exceptions
+
+Custom errors are provided which can be used to control the execution flow in case something went wrong. All those errors can be raised in protocol methods and will be caught by the agent.  
+By default agent will retry three times and then re-raise an exception if it's still not resolved. All passed arguments are automatically handled and the values are reverted when needed.
+All errors accept an optional `str` message. There are following errors ordered by increasing broadness:
+
+1. `ComponentEndpointError`: A single endpoint method failed to execute. Agent will retry the execution of this endpoint on the component.
+2. `EndpointPipelineError`: A pipeline failed to execute. Agent will retry the execution of the endpoint for all components.
+3. `ComponentSystemError`: Multiple pipelines failed.
+
+**Example**
+
+```py
+from autogpt.agents.components import ComponentEndpointError
+from autogpt.agents.protocols import MessageProvider
+
+# Example of raising an error
+class MyComponent(MessageProvider):
+    def get_messages(self) -> Iterator[ChatMessage]:
+        # This will cause the component to always fail 
+        # and retry 3 times before re-raising the exception
+        raise ComponentEndpointError("Endpoint error!")
+```
diff --git a/docs/content/AutoGPT/component agent/creating-components.md b/docs/content/AutoGPT/component agent/creating-components.md
new file mode 100644
index 000000000..d06714beb
--- /dev/null
+++ b/docs/content/AutoGPT/component agent/creating-components.md
@@ -0,0 +1,235 @@
+# Creating Components
+
+## The minimal component
+
+Components can be used to implement various functionalities like providing messages to the prompt, executing code, or interacting with external services.
+
+*Component* is a class that inherits from `AgentComponent` OR implements one or more *protocols*. Every *protocol* inherits `AgentComponent`, so your class automatically becomes a *component* once you inherit any *protocol*.
+
+```py
+class MyComponent(AgentComponent):
+    pass
+```
+
+This is already a valid component, but it doesn't do anything yet. To add some functionality to it, you need to implement one or more *protocols*.
+
+Let's create a simple component that adds "Hello World!" message to the agent's prompt. To do this we need to implement `MessageProvider` *protocol* in our component. `MessageProvider` is an interface with `get_messages` method:
+
+```py
+# No longer need to inherit AgentComponent, because MessageProvider already does it
+class HelloComponent(MessageProvider):
+    def get_messages(self) -> Iterator[ChatMessage]:
+        yield ChatMessage.user("Hello World!")
+```
+
+Now we can add our component to an existing agent or create a new Agent class and add it there:
+
+```py
+class MyAgent(Agent):
+    self.hello_component = HelloComponent()
+```
+
+`get_messages` will called by the agent each time it needs to build a new prompt and the yielded messages will be added accordingly.  
+
+## Passing data to and between components
+
+Since components are regular classes you can pass data (including other components) to them via the `__init__` method.
+For example we can pass a config object and then retrieve an API key from it when needed:
+
+```py
+class ConfigurableComponent(MessageProvider):
+    def __init__(self, config: Config):
+        self.config = config
+
+    def get_messages(self) -> Iterator[ChatMessage]:
+        if self.config.openai_credentials.api_key:
+            yield ChatMessage.system("API key found!")
+        else:
+            yield ChatMessage.system("API key not found!")
+```
+
+!!! note
+    Component-specific configuration handling isn't implemented yet.
+
+## Providing commands
+
+To extend what an agent can do, you need to provide commands using `CommandProvider` protocol. For example to allow agent to multiply two numbers, you can create a component like this:
+
+```py
+class MultiplicatorComponent(CommandProvider):
+    def get_commands(self) -> Iterator[Command]:
+        # Yield the command so the agent can use it
+        yield self.multiply
+
+    @command(
+    parameters={
+        "a": JSONSchema(
+            type=JSONSchema.Type.INTEGER,
+            description="The first number",
+            required=True,
+        ),
+        "b": JSONSchema(
+            type=JSONSchema.Type.INTEGER,
+            description="The second number",
+            required=True,
+        )})
+    def multiply(self, a: int, b: int) -> str:
+        """
+        Multiplies two numbers.
+        
+        Args:
+            a: First number
+            b: Second number
+
+        Returns:
+            Result of multiplication
+        """
+        return str(a * b)
+```
+
+To learn more about commands see [🛠️ Commands](./commands.md).
+
+## Prompt structure
+
+After components provided all necessary data, the agent needs to build the final prompt that will be send to a llm.
+Currently, `PromptStrategy` (*not* a protocol) is responsible for building the final prompt.
+If you want to change the way the prompt is built, you need to create a new `PromptStrategy` class, and then call relavant methods in your agent class.
+You can have a look at the default strategy used by the AutoGPT Agent: [OneShotAgentPromptStrategy](../../../../autogpts/autogpt/autogpt/agents/prompt_strategies/one_shot.py), and how it's used in the [Agent](../../../../autogpts/autogpt/autogpt/agents/agent.py) (search for `self.prompt_strategy`).
+
+## Example `UserInteractionComponent`
+
+Let's create a slighlty simplified version of the component that is used by the built-in agent.
+It gives an ability for the agent to ask user for input in the terminal.
+
+1. Create a class for the component that inherits from `CommandProvider`.
+```py
+class MyUserInteractionComponent(CommandProvider):
+    """Provides commands to interact with the user."""
+    pass
+```
+
+2. Implement command method that will ask user for input and return it.
+```py
+def ask_user(self, question: str) -> str:
+    """If you need more details or information regarding the given goals,
+    you can ask the user for input."""
+    print(f"\nQ: {question}")
+    resp = input("A:")
+    return f"The user's answer: '{resp}'"
+```
+
+3. The command needs to be decorated with `@command`.
+```py
+@command(
+    parameters={
+        "question": JSONSchema(
+            type=JSONSchema.Type.STRING,
+            description="The question or prompt to the user",
+            required=True,
+        )
+    },
+)
+def ask_user(self, question: str) -> str:
+    """If you need more details or information regarding the given goals,
+    you can ask the user for input."""
+    print(f"\nQ: {question}")
+    resp = input("A:")
+    return f"The user's answer: '{resp}'"
+```
+
+4. We need to implement `CommandProvider`'s `get_commands` method to yield the command.
+```py
+def get_commands(self) -> Iterator[Command]:
+    yield self.ask_user
+```
+
+5. Since agent isn't always running in the terminal or interactive mode, we need to disable this component by setting `self._enabled` when it's not possible to ask for user input.
+```py
+def __init__(self, config: Config):
+    self.config = config
+    self._enabled = not config.noninteractive_mode
+```
+
+The final component should look like this:
+
+```py
+# 1.
+class MyUserInteractionComponent(CommandProvider):
+    """Provides commands to interact with the user."""
+
+    # We pass config to check if we're in noninteractive mode
+    def __init__(self, config: Config):
+        self.config = config
+        # 5.
+        self._enabled = not config.noninteractive_mode
+
+    # 4.
+    def get_commands(self) -> Iterator[Command]:
+        # Yielding the command so the agent can use it
+        # This won't be yielded if the component is disabled
+        yield self.ask_user
+
+    # 3.
+    @command(
+        # We need to provide a schema for ALL the command parameters
+        parameters={
+            "question": JSONSchema(
+                type=JSONSchema.Type.STRING,
+                description="The question or prompt to the user",
+                required=True,
+            )
+        },
+    )
+    # 2.
+    # Command name will be its method name and description will be its docstring
+    def ask_user(self, question: str) -> str:
+        """If you need more details or information regarding the given goals,
+        you can ask the user for input."""
+        print(f"\nQ: {question}")
+        resp = input("A:")
+        return f"The user's answer: '{resp}'"
+```
+
+Now if we want to use our user interaction *instead of* the default one we need to somehow remove the default one (if our agent inherits from `Agent` the default one is inherited) and add our own. We can simply override the `user_interaction` in `__init__` method:
+
+```py
+class MyAgent(Agent):
+    def __init__(
+        self,
+        settings: AgentSettings,
+        llm_provider: ChatModelProvider,
+        file_storage: FileStorage,
+        legacy_config: Config,
+    ):
+        # Call the parent constructor to bring in the default components
+        super().__init__(settings, llm_provider, file_storage, legacy_config)
+        # Disable the default user interaction component by overriding it
+        self.user_interaction = MyUserInteractionComponent()
+```
+
+Alternatively we can disable the default component by setting it to `None`:
+
+```py
+class MyAgent(Agent):
+    def __init__(
+        self,
+        settings: AgentSettings,
+        llm_provider: ChatModelProvider,
+        file_storage: FileStorage,
+        legacy_config: Config,
+    ):
+        # Call the parent constructor to bring in the default components
+        super().__init__(settings, llm_provider, file_storage, legacy_config)
+        # Disable the default user interaction component
+        self.user_interaction = None
+        # Add our own component
+        self.my_user_interaction = MyUserInteractionComponent(legacy_config)
+```
+
+## Learn more
+
+The best place to see more examples is to look at the built-in components in the [autogpt/components](../../../../autogpts/autogpt/autogpt/components/) and [autogpt/commands](../../../../autogpts/autogpt/autogpt/commands/) directories.
+
+Guide on how to extend the built-in agent and build your own: [🤖 Agents](./agents.md)  
+Order of some components matters, see [🧩 Components](./components.md) to learn more about components and how they can be customized.  
+To see built-in protocols with accompanying examples visit [⚙️ Protocols](./protocols.md).
diff --git a/docs/content/AutoGPT/component agent/introduction.md b/docs/content/AutoGPT/component agent/introduction.md
new file mode 100644
index 000000000..954b67770
--- /dev/null
+++ b/docs/content/AutoGPT/component agent/introduction.md
@@ -0,0 +1,17 @@
+# Component Agents
+
+This guide explains the component-based architecture of AutoGPT agents. It's a new way of building agents that is more flexible and easier to extend. Components replace some agent's logic and plugins with a more modular and composable system.
+
+Agent is composed of *components*, and each *component* implements a range of *protocols* (interfaces), each one providing a specific functionality, e.g. additional commands or messages. Each *protocol* is handled in a specific order, defined by the agent. This allows for a clear separation of concerns and a more modular design.
+
+This system is simple, flexible, requires basically no configuration, and doesn't hide any data - anything can still be passed or accessed directly from or between components.
+
+### Definitions & Guides
+
+See [Creating Components](./creating-components.md) to get started! Or you can explore the following topics in detail:
+
+- [🧩 Component](./components.md): a class that implements one or more *protocols*. It can be added to an agent to provide additional functionality. See what's already provided in [Built-in Components](./built-in-components.md).
+- [⚙️ Protocol](./protocols.md): an interface that defines a set of methods that a component must implement. Protocols are used to group related functionality.
+- [🛠️ Command](./commands.md): enable *agent* to interact with user and tools.
+- [🤖 Agent](./agents.md): a class that is composed of components. It's responsible for executing pipelines and managing the components.
+- **Pipeline**: a sequence of method calls on components. Pipelines are used to execute a series of actions in a specific order. As of now there's no formal class for a pipeline, it's just a sequence of method calls on components. There are two default pipelines implemented in the default agent: `propose_action` and `execute`. See [🤖 Agent](./agents.md) to learn more.
diff --git a/docs/content/AutoGPT/component agent/protocols.md b/docs/content/AutoGPT/component agent/protocols.md
new file mode 100644
index 000000000..b3d0be8cf
--- /dev/null
+++ b/docs/content/AutoGPT/component agent/protocols.md
@@ -0,0 +1,166 @@
+# ⚙️ Protocols
+
+Protocols are *interfaces* implemented by [Components](./components.md) used to group related functionality. Each protocol needs to be handled explicitly by the agent at some point of the execution. We provide a comprehensive list of built-in protocols that are already handled in the built-in `Agent`, so when you inherit from the base agent all built-in protocols will work!
+
+**Protocols are listed in the order of the default execution.**
+
+## Order-independent protocols
+
+Components implementing exclusively order-independent protocols can added in any order, including in-between ordered protocols.
+
+### `DirectiveProvider`
+
+Yields constraints, resources and best practices for the agent. This has no direct impact on other protocols; is purely informational and will be passed to a llm when the prompt is built.
+
+```py
+class DirectiveProvider(AgentComponent):
+    def get_constraints(self) -> Iterator[str]:
+        return iter([])
+
+    def get_resources(self) -> Iterator[str]:
+        return iter([])
+
+    def get_best_practices(self) -> Iterator[str]:
+        return iter([])
+```
+
+**Example** A web-search component can provide a resource information. Keep in mind that this actually doesn't allow the agent to access the internet. To do this a relevant `Command` needs to be provided.
+
+```py
+class WebSearchComponent(DirectiveProvider):
+    def get_resources(self) -> Iterator[str]:
+        yield "Internet access for searches and information gathering."
+    # We can skip "get_constraints" and "get_best_practices" if they aren't needed
+```
+
+### `CommandProvider`
+
+Provides a command that can be executed by the agent.
+
+```py
+class CommandProvider(AgentComponent):
+    def get_commands(self) -> Iterator[Command]:
+        ...
+```
+
+The easiest way to provide a command is to use `command` decorator on a component method and then yield the method. Each command needs a name, description and a parameter schema using `JSONSchema`. By default method name is used as a command name, and first part of docstring for the description (before `Args:` or `Returns:`) and schema can be provided in the decorator.
+
+**Example** Calculator component that can perform multiplication. Agent is able to call this command if it's relevant to a current task and will see the returned result.
+
+```py
+from autogpt.agents.components import Component
+from autogpt.agents.protocols import CommandProvider
+from autogpt.core.utils.json_schema import JSONSchema
+from autogpt.utils.command_decorator import command
+
+
+class CalculatorComponent(CommandProvider):
+    get_commands(self) -> Iterator[Command]:
+        yield self.multiply
+
+    @command(parameters={
+            "a": JSONSchema(
+                type=JSONSchema.Type.INTEGER,
+                description="The first number",
+                required=True,
+            ),
+            "b": JSONSchema(
+                type=JSONSchema.Type.INTEGER,
+                description="The second number",
+                required=True,
+            )})
+    def multiply(self, a: int, b: int) -> str:
+        """
+        Multiplies two numbers.
+        
+        Args:
+            a: First number
+            b: Second number
+
+        Returns:
+            Result of multiplication
+        """
+        return str(a * b)
+```
+
+The agent will be able to call this command, named `multiply` with two arguments and will receive the result. The command description will be: `Multiplies two numbers.`
+
+To learn more about commands see [🛠️ Commands](./commands.md).
+
+## Order-dependent protocols
+
+The order of components implementing order-dependent protocols is important.
+Some components may depend on the results of components before them.
+
+### `MessageProvider`
+
+Yields messages that will be added to the agent's prompt. You can use either `ChatMessage.user()`: this will interpreted as a user-sent message or `ChatMessage.system()`: that will be more important.
+
+```py
+class MessageProvider(AgentComponent):
+    def get_messages(self) -> Iterator[ChatMessage]:
+        ...
+```
+
+**Example** Component that provides a message to the agent's prompt.
+
+```py
+class HelloComponent(MessageProvider):
+    def get_messages(self) -> Iterator[ChatMessage]:
+        yield ChatMessage.user("Hello World!")
+```
+
+### `AfterParse`
+
+Protocol called after the response is parsed.
+
+```py
+class AfterParse(AgentComponent):
+    def after_parse(self, response: ThoughtProcessOutput) -> None:
+        ...
+```
+
+**Example** Component that logs the response after it's parsed.
+
+```py
+class LoggerComponent(AfterParse):
+    def after_parse(self, response: ThoughtProcessOutput) -> None:
+        logger.info(f"Response: {response}")
+```
+
+### `ExecutionFailure` 
+
+Protocol called when the execution of the command fails.
+
+```py
+class ExecutionFailure(AgentComponent):
+    @abstractmethod
+    def execution_failure(self, error: Exception) -> None:
+        ...
+```
+
+**Example** Component that logs the error when the command fails.
+
+```py
+class LoggerComponent(ExecutionFailure):
+    def execution_failure(self, error: Exception) -> None:
+        logger.error(f"Command execution failed: {error}")
+```
+
+### `AfterExecute`
+
+Protocol called after the command is successfully executed by the agent.
+
+```py
+class AfterExecute(AgentComponent):
+    def after_execute(self, result: ActionResult) -> None:
+        ...
+```
+
+**Example** Component that logs the result after the command is executed.
+
+```py
+class LoggerComponent(AfterExecute):
+    def after_execute(self, result: ActionResult) -> None:
+        logger.info(f"Result: {result}")
+```
\ No newline at end of file
diff --git a/docs/content/AutoGPT/configuration/options.md b/docs/content/AutoGPT/configuration/options.md
index dd0291bfa..c84d70cfb 100644
--- a/docs/content/AutoGPT/configuration/options.md
+++ b/docs/content/AutoGPT/configuration/options.md
@@ -11,7 +11,7 @@ Configuration is controlled through the `Config` object. You can set configurati
 - `BROWSE_CHUNK_MAX_LENGTH`: When browsing website, define the length of chunks to summarize. Default: 3000
 - `BROWSE_SPACY_LANGUAGE_MODEL`: [spaCy language model](https://spacy.io/usage/models) to use when creating chunks. Default: en_core_web_sm
 - `CHAT_MESSAGES_ENABLED`: Enable chat messages. Optional
-- `DISABLED_COMMAND_CATEGORIES`: Command categories to disable. Command categories are Python module names, e.g. autogpt.commands.execute_code. See the directory `autogpt/commands` in the source for all command modules. Default: None
+- `DISABLED_COMMANDS`: Commands to disable. Use comma separated names of commands. See the list of commands from built-in components [here](../component%20agent/components.md). Default: None
 - `ELEVENLABS_API_KEY`: ElevenLabs API Key. Optional.
 - `ELEVENLABS_VOICE_ID`: ElevenLabs Voice ID. Optional.
 - `EMBEDDING_MODEL`: LLM Model to use for embedding tasks. Default: `text-embedding-3-small`
diff --git a/docs/content/AutoGPT/plugins.md b/docs/content/AutoGPT/plugins.md
deleted file mode 100644
index 4733a8b3d..000000000
--- a/docs/content/AutoGPT/plugins.md
+++ /dev/null
@@ -1,20 +0,0 @@
-## Plugins
-
-⚠️💀 **WARNING** 💀⚠️: Review the code of any plugin you use thoroughly, as plugins can execute any Python code, potentially leading to malicious activities, such as stealing your API keys.
-
-To configure plugins, you can create or edit the `plugins_config.yaml` file in the root directory of AutoGPT. This file allows you to enable or disable plugins as desired. For specific configuration instructions, please refer to the documentation provided for each plugin. The file should be formatted in YAML. Here is an example for your reference:
-
-```yaml
-plugin_a:
-  config:
-    api_key: my-api-key
-  enabled: false
-plugin_b:
-  config: {}
-  enabled: true
-```
-
-See our [Plugins Repo](https://github.com/Significant-Gravitas/Auto-GPT-Plugins) for more info on how to install all the amazing plugins the community has built!
-
-Alternatively, developers can use the [AutoGPT Plugin Template](https://github.com/Significant-Gravitas/Auto-GPT-Plugin-Template) as a starting point for creating your own plugins.
-
diff --git a/docs/content/AutoGPT/usage.md b/docs/content/AutoGPT/usage.md
index 09447f03d..6e7b81587 100644
--- a/docs/content/AutoGPT/usage.md
+++ b/docs/content/AutoGPT/usage.md
@@ -214,16 +214,18 @@ To print out debug logs:
 ./autogpt.sh --debug
 ```
 
-## Disabling Command Categories
+## Disabling Commands
 
-If you want to selectively disable some command groups, you can use the
-`DISABLED_COMMAND_CATEGORIES` config in your `.env`. You can find the list of available
-categories [here][command categories].
+The best way to disable commands is to disable or remove the [component][components] that provides them.
+However, if you want to selectively disable some commands, you can use the `DISABLED_COMMANDS` config in your `.env`.
+Put the names of the commands you want to disable, separated by commas.
+You can find the list of commands in built-in components [here][commands].
 
-For example, to disable coding related features, set it to the value below:
+For example, to disable python coding features, set it to the value below:
 
 ```ini
-DISABLED_COMMAND_CATEGORIES=autogpt.commands.execute_code
+DISABLED_COMMANDS=execute_python_code,execute_python_file
 ```
 
-[command categories]: https://github.com/Significant-Gravitas/AutoGPT/blob/master/autogpts/autogpt/autogpt/commands/__init__.py
+[components]: ./component%20agent/components.md
+[commands]: ./component%20agent/built-in-components.md
\ No newline at end of file
diff --git a/docs/content/imgs/modular-pipeline.png b/docs/content/imgs/modular-pipeline.png
new file mode 100644
index 000000000..9f9c58047
--- /dev/null
+++ b/docs/content/imgs/modular-pipeline.png