Skip to content

Conversation

Seol-JY
Copy link
Contributor

@Seol-JY Seol-JY commented Jul 20, 2025

Fixes GH-3403

Add ConcurrentHashMap-based caching to eliminate repeated schema generation for identical method signatures in tool invocations.

  • Cache tool definitions by Method instance in ToolDefinitions
  • Cache JSON schemas with method signature + options as key
  • Add comprehensive test coverage for caching behavior

Performance improvement: Eliminates CPU overhead from repeated schema generation in function calling applications.

…ction calling

Fixes spring-projectsGH-3403 (spring-projects#3403)

Add ConcurrentHashMap-based caching to eliminate repeated schema
generation for identical method signatures in tool invocations.

* Cache tool definitions by Method instance in ToolDefinitions
* Cache JSON schemas with method signature + options as key
* Include comprehensive test coverage for caching behavior

Performance improvement: Eliminates CPU overhead from repeated
schema generation in function calling applications.

Signed-off-by: Seol-JY <wlsdud5654@gmail.com>
Fix formatting violations detected by spring-javaformat-maven-plugin

Signed-off-by: Seol-JY <wlsdud5654@gmail.com>
@YunKuiLu
Copy link
Contributor

Can we also cache ToolCallback[] in MethodToolCallbackProvider ? 🤔

Such as:

public final class MethodToolCallbackProvider implements ToolCallbackProvider {
    @Override
    public ToolCallback[] getToolCallbacks() {
    if (this.cachedToolCallbacks == null) {
	    synchronized (this) {
		    if (this.cachedToolCallbacks == null) {
			    var toolCallbacks = this.toolObjects.stream()
				    .map(toolObject -> Stream
					    .of(ReflectionUtils.getDeclaredMethods(AopUtils.isAopProxy(toolObject)
							    ? AopUtils.getTargetClass(toolObject) : toolObject.getClass()))
					    .filter(this::isToolAnnotatedMethod)
					    .filter(toolMethod -> !isFunctionalType(toolMethod))
					    .filter(ReflectionUtils.USER_DECLARED_METHODS::matches)
					    .map(toolMethod -> MethodToolCallback.builder()
						    .toolDefinition(ToolDefinitions.from(toolMethod))
						    .toolMetadata(ToolMetadata.from(toolMethod))
						    .toolMethod(toolMethod)
						    .toolObject(toolObject)
						    .toolCallResultConverter(ToolUtils.getToolCallResultConverter(toolMethod))
						    .build())
					    .toArray(ToolCallback[]::new))
				    .flatMap(Stream::of)
				    .toArray(ToolCallback[]::new);
    
			    validateToolCallbacks(toolCallbacks);
    
			    this.cachedToolCallbacks = toolCallbacks;
		    }
	    }
    }
    return this.cachedToolCallbacks;
    }
}

@Seol-JY
Copy link
Contributor Author

Seol-JY commented Jul 22, 2025

Hi @YunKuiLu,
Thanks for the suggestion! While the ToolCallback caching could be beneficial, I think it's outside the scope of this PR, which specifically targets JSON schema generation caching (GH-3403).
Also, I'm not entirely sure how frequently getToolCallbacks() is called in practice - if it's mainly during initialization, the performance gain might be limited.
Perhaps we could explore this in a separate issue?

@Seol-JY
Copy link
Contributor Author

Seol-JY commented Aug 2, 2025

Hi @ilayaperumalg , when you have a moment, could you please review my PR?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Consider caching JSON schema generation in function calling
2 participants