Enabled function calling breaks shell command generation #867
Description
Describe the bug
If function calling is enabled, shell command generation breaks. Affected are aichat -r %shell% and aichat -e invocations.
aichat -r %shell%:
❯ aichat -r %shell% what is in the current folder
Call fs_ls {"path":"."}
ls
PS C:\Users\User> aichat -r %shell% what is in the current folder
{"command":"Get-ChildItem"}
Call execute_command {"command":"Get-ChildItem"}
Invalid JSON data
Tool call exit with 1
I have a use case where I need to generate shell commands and directly use them, but the output contains additional meta information about the called function: Call fs_ls {"path":"."}. No idea what went wrong with the PowerShell one.
aichat -e:
❯ aichat -e what is in the current folder
?
> ✅ Execute
🔄️ Revise
📖 Explain
❌ Cancel
[↑↓ to move, enter to select, type to filter]
PS C:\Users\User> aichat -e what is in the current folder
? {"command":"Get-ChildItem"}
> ✅ Execute
🔄️ Revise
📖 Explain
❌ Cancel
If a function call happens during the invocation of aichat -e, the resulting line buffer is empty or contains something that is not executable.
To Reproduce
- globally enable function calling
- invoke
aichat -r %shell%oraichat -ewith a prompt which is likely to trigger function calling
Expected behavior
aichat -r %shell%:
❯ aichat -r %shell% what is in the current folder
ls
=> the output contains only the command without noise
aichat -e:
❯ aichat -e what is in the current folder
? ls
> ✅ Execute
🔄️ Revise
📖 Explain
❌ Cancel
[↑↓ to move, enter to select, type to filter]
=> the line buffer contains something useful
Configuration
model gpt-4o:gpt-4o-2024-08-06
max_output_tokens -
temperature 0.1
top_p -
dry_run false
stream true
save false
keybindings emacs
wrap no
wrap_code false
save_session false
compress_threshold 4000
function_calling true
use_tools execute,fs,info,request,search
rag_reranker_model -
rag_top_k 4
highlight true
light_theme false
config_file C:\Users\User\AppData\Roaming\aichat\config.yaml
roles_dir C:\Users\User\AppData\Roaming\aichat\roles
env_file C:\Users\User\AppData\Roaming\aichat\.env
functions_dir C:\Users\User\AppData\Roaming\aichat\functions
rags_dir C:\Users\User\AppData\Roaming\aichat\rags
sessions_dir C:\Users\User\AppData\Roaming\aichat\sessions
messages_file C:\Users\User\AppData\Roaming\aichat\messages.md
Environment (please complete the following information):
- os version: Windows 11
- aichat version: 0.21.1
- terminal version: Cygwin Fish / PowerShell 7.4.5
Additional context
Possible solutions might be:
- a (global / per role) option to suppress the function calling output
- a per role option to disable or override globally enabled functions
- maybe the shell role system prompt can be improved a bit so that even sloppily worded prompts like in this example don't lead to unintended behavior. Instead of giving me the correct command, the LLM tried to execute the command itself.