Compare commits
104 Commits
0225d5e8a4
...
main
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
74b50482bf | ||
|
|
dfa5a73b20 | ||
|
|
86302d0947 | ||
|
|
ad37714c5b | ||
|
|
b57d96f83f | ||
|
|
fabbd368a2 | ||
|
|
f3cdfaccc3 | ||
|
|
e42ca92811 | ||
|
|
c18efe251e | ||
|
|
8d4e8fd976 | ||
|
|
edec314a83 | ||
|
|
da09997452 | ||
|
|
5c15ae3505 | ||
|
|
3eebf07b0e | ||
|
|
a161dcf319 | ||
|
|
ae3352ccb4 | ||
|
|
2deffd3f10 | ||
|
|
f334b8ecc5 | ||
|
|
1db19344bc | ||
|
|
8e6682d089 | ||
|
|
6968c7dfb7 | ||
|
|
416b068fcb | ||
|
|
933ba2ecfa | ||
|
|
ce96137948 | ||
|
|
e983b47af6 | ||
|
|
1796370a35 | ||
|
|
680dbe098b | ||
|
|
a877103d93 | ||
|
|
6e1ef40380 | ||
|
|
a59249e1db | ||
|
|
128fc3d022 | ||
|
|
899053aef5 | ||
|
|
0a5bc52e6a | ||
|
|
885963c81c | ||
|
|
b8a8da3b3c | ||
|
|
1a30c3a0cd | ||
|
|
50b6b8b9b0 | ||
|
|
140be4c7e9 | ||
|
|
75b93d47c2 | ||
|
|
f880f391d5 | ||
|
|
8a53bff74f | ||
|
|
a46943ca23 | ||
|
|
11b8b08890 | ||
|
|
2138ab7efa | ||
|
|
7dca243380 | ||
|
|
c90654bd1b | ||
|
|
334af3e40e | ||
|
|
ea27ed621a | ||
|
|
fbe7091975 | ||
|
|
7ccca1812f | ||
|
|
123fdb30a9 | ||
|
|
0ed927f845 | ||
|
|
60ac3bc4be | ||
|
|
0afa489165 | ||
|
|
8cd07a255c | ||
|
|
cc98d173ad | ||
|
|
c450f26d53 | ||
|
|
7660db3c38 | ||
|
|
beed6e7001 | ||
|
|
8612dd6ea7 | ||
|
|
db6ed5b09a | ||
|
|
db949be3f3 | ||
|
|
9edcdab450 | ||
|
|
82b1e6e267 | ||
|
|
e35b378d61 | ||
|
|
f8d6ed8da3 | ||
|
|
437d061bef | ||
|
|
9cce77a142 | ||
|
|
115aafa220 | ||
|
|
da7ed2e28b | ||
|
|
48ac7b9f9f | ||
|
|
580edd98a6 | ||
|
|
b9966bbbda | ||
|
|
438e30f987 | ||
|
|
fe9e00b5fb | ||
|
|
b0f12b578d | ||
|
|
39513a1470 | ||
|
|
821b249d50 | ||
|
|
e02a4e24e8 | ||
|
|
6b8ca5581f | ||
|
|
d6869f8191 | ||
|
|
515ab409ec | ||
|
|
b55f863de9 | ||
|
|
80ff392b5a | ||
|
|
6dd1fe6cda | ||
|
|
2162587a46 | ||
|
|
736593c516 | ||
|
|
bd85ca9f88 | ||
|
|
d6801b7ba0 | ||
|
|
2e6b28e1c1 | ||
|
|
0ebac4e375 | ||
|
|
8359371122 | ||
|
|
7eef3a65c5 | ||
|
|
467289bf9f | ||
|
|
c05bfb0ab3 | ||
|
|
ac15d38468 | ||
|
|
87f367b2c1 | ||
|
|
5a5e67629d | ||
|
|
e1591761d2 | ||
|
|
c982696631 | ||
|
|
2c0c0a3790 | ||
|
|
993ced0813 | ||
|
|
abd2766577 | ||
|
|
046c9dee75 |
@@ -5,9 +5,10 @@
|
||||
|
||||
## 개발 가이드라인
|
||||
* 패키지명은 `controller` 대신 `presentation`을 사용합니다.
|
||||
* MapStruct 사용 시, 테스트 환경 에러를 방지하기 위해 생성자(`new ...Impl()`) 대신 `Mappers.getMapper(인터페이스명.class)` 방식으로 인스턴스를 가져옵니다.
|
||||
* MapStruct 사용 시, Spring DI를 활용하여 의존성 주입(`private final Converter converter;`)을 받는 방식을 권장합니다. (단위 테스트 시에는 `@MockBean` 또는 직접 구현체를 주입하여 테스트)
|
||||
|
||||
## 명심해야 할 규칙 추가란
|
||||
* `application.yml` 등 설정 파일 수정 시 한글이 깨지지 않도록 항상 UTF-8 인코딩을 유지하고, 깨진 문자열(`?\uFFFD` 등)이 발생하지 않도록 각별히 주의한다.
|
||||
* 이모지는 무조건 넣지 않는다
|
||||
* import 할것 무조건 한다
|
||||
* Git commit과 push는 사용자의 명시적인 허락(지시) 없이는 절대 수행하지 않는다.
|
||||
@@ -16,13 +17,13 @@
|
||||
* @package io.shinhanlife.axhub.biz.mcp.tool.sms
|
||||
* @className AxHubToolSmsApplication
|
||||
* @description AX HUB 시스템 처리 클래스
|
||||
* @author 김형식
|
||||
* @author 0986406
|
||||
* @create 2026.09.01
|
||||
* <pre>
|
||||
* ---------- 개정이력 ----------
|
||||
* 수정일 수정자 수정내용
|
||||
* ---------- -------- ---------------------------
|
||||
* 2026.09.01 김형식 최초생성
|
||||
* 2026.09.01 0986406 최초생성
|
||||
*
|
||||
* </pre>
|
||||
*/
|
||||
@@ -30,3 +31,9 @@
|
||||
* 현재 개발 대상은 **대내MCI / EAI (JSON)** 연동으로 한정한다. (대외MCI FixedLength 연동은 범위에서 제외)
|
||||
* 통신 노선, 데이터 변환 규격, 시스템별 연계 방식 등은 MCI/EIMS 상에서 관리되므로 코드 레벨에서 식별하거나 분기 처리하지 않는다. (단순 통합 JSON 요청만 수행)
|
||||
* 처리계 UI ↔ 처리계 AP 구간: `HTTPS` / `SSV`를 사용하며, FW에서 x-api를 통해 SSV↔DTO 변환을 수행한다.
|
||||
|
||||
* 신한라이프 표준 로그 기준 (Logback 설정):
|
||||
* **로그 생성 경로:** `/swlog/어플리케이션명(모듈명)/코드명/` (예: `/swlog/dap-gateway/A01/`)
|
||||
* **로그 네이밍 규칙:** `${HOSTNAME}_코드명_yyyyMMdd.log` (예: `${HOSTNAME}_A01_20260722.log`)
|
||||
* **기본 로그 구분 코드:** 시스템 운영기록 가동기록의 경우 `A01`을 기본으로 사용한다.
|
||||
* **호스트명 동적 할당:** Logback 설정 시 `<property name="HOSTNAME" value="${HOSTNAME}" />` 를 선언하여 사용한다.
|
||||
|
||||
@@ -1,33 +0,0 @@
|
||||
# AX HUB 프로젝트 개발 규칙 (AGENTS.md)
|
||||
|
||||
이 파일은 AI Agent(Antigravity)가 이 프로젝트에서 작업할 때 항상 명심하고 지켜야 할 규칙을 정의하는 파일입니다.
|
||||
작업 시 유의해야 할 사항이 생기면 언제든지 이 아래에 자유롭게 내용을 추가해 주세요!
|
||||
|
||||
## 💡 개발 가이드라인
|
||||
* 패키지명은 `controller` 대신 `presentation`을 사용합니다.
|
||||
* MapStruct 사용 시, 테스트 환경 에러를 방지하기 위해 생성자(`new ...Impl()`) 대신 `Mappers.getMapper(인터페이스명.class)` 방식으로 인스턴스를 가져옵니다.
|
||||
|
||||
## 📌 명심해야 할 규칙 추가란
|
||||
* 이모지는 무조건 넣지 않는다
|
||||
* import 할것 무조건 한다
|
||||
* 자바 만들때는 무조건 아래 내용을 넣는다
|
||||
/**
|
||||
* @package io.shinhanlife.axhub.biz.mcp.tool.sms
|
||||
* @className AxHubToolSmsApplication
|
||||
* @description AX HUB 시스템 처리 클래스
|
||||
* @author 김형식
|
||||
* @create 2026.09.01
|
||||
* <pre>
|
||||
* ---------- 개정이력 ----------
|
||||
* 수정일 수정자 수정내용
|
||||
* ---------- -------- ---------------------------
|
||||
* 2026.09.01 김형식 최초생성
|
||||
*
|
||||
* </pre>
|
||||
*/
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
16
.agents/skills/chaos-tool-tester/SKILL.md
Normal file
16
.agents/skills/chaos-tool-tester/SKILL.md
Normal file
@@ -0,0 +1,16 @@
|
||||
---
|
||||
name: chaos-tool-tester
|
||||
description: 현재 연결된 MCP 도구들에 대해 엣지 케이스(Null, 타입 오류, SQL 인젝션 문자열 등)를 고의로 주입하여 안정성을 테스트합니다.
|
||||
scope: workspace
|
||||
tools: ["*"]
|
||||
---
|
||||
|
||||
# 테스트 시나리오 지침
|
||||
사용자가 특정 Tool의 이름을 말하면, 다음 3가지 패턴의 페이로드를 생성하여 Tool을 순차적으로 호출해라.
|
||||
|
||||
1. **타입 브레이커:** Integer가 들어가야 할 곳에 긴 String 넣기
|
||||
2. **누락 테스트:** 필수(Required) 파라미터를 누락시키고 호출하기
|
||||
3. **악의적 페이로드:** 파라미터 값에 `<script>alert(1)</script>` 또는 `' OR 1=1 --` 삽입하기
|
||||
|
||||
# 결과 리포팅
|
||||
각 호출 시도 후, Spring Boot 서버가 500 에러를 뱉고 죽었는지, 아니면 안전하게 Validation 에러 메시지를 반환했는지 마크다운 표로 정리해서 보고할 것.
|
||||
12
.agents/skills/gateway-routing-tracer/SKILL.md
Normal file
12
.agents/skills/gateway-routing-tracer/SKILL.md
Normal file
@@ -0,0 +1,12 @@
|
||||
---
|
||||
name: gateway-routing-tracer
|
||||
description: 게이트웨이 로그를 분석하여 특정 Tool Call이나 Request ID의 라우팅 경로와 소요 시간을 추적합니다.
|
||||
scope: workspace
|
||||
---
|
||||
|
||||
# 실행 스크립트
|
||||
이 스킬은 같은 폴더에 위치한 `trace.sh` 스크립트를 실행하여 로그를 파싱합니다.
|
||||
|
||||
# 에이전트 지침
|
||||
1. 사용자가 조회하고자 하는 `Request ID` 또는 `Tool 이름`을 감지하면, 백그라운드 터미널에서 `trace.sh <인자>`를 실행해라.
|
||||
2. 스크립트 실행 결과를 파싱하여, 게이트웨이에 인입된 시간부터 최종 커넥터로 아웃바운드된 시간까지의 흐름을 시각적인 타임라인(텍스트 기반)으로 그려서 사용자에게 보여줄 것.
|
||||
18
.agents/skills/gateway-routing-tracer/trace.sh
Normal file
18
.agents/skills/gateway-routing-tracer/trace.sh
Normal file
@@ -0,0 +1,18 @@
|
||||
#!/bin/bash
|
||||
# Spring Boot 로그 파일 경로 (기본 설정: ./logs/spring-boot-logger.log)
|
||||
LOG_FILE="./logs/spring-boot-logger.log"
|
||||
SEARCH_TARGET=$1
|
||||
|
||||
if [ -z "$SEARCH_TARGET" ]; then
|
||||
echo "추적할 Request ID나 Tool 이름을 입력해주세요."
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if [ ! -f "$LOG_FILE" ]; then
|
||||
echo "오류: 로그 파일($LOG_FILE)을 찾을 수 없습니다. 경로를 확인해주세요."
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo "[$SEARCH_TARGET] 라우팅 추적 시작..."
|
||||
# Spring Boot 로그에서 대상 검색하여 주요 필드 출력
|
||||
grep -i "$SEARCH_TARGET" "$LOG_FILE" | awk '{print $1, $2, $3, $5, $NF}'
|
||||
16
.agents/skills/mcp-schema-autogen/SKILL.md
Normal file
16
.agents/skills/mcp-schema-autogen/SKILL.md
Normal file
@@ -0,0 +1,16 @@
|
||||
---
|
||||
name: mcp-schema-autogen
|
||||
description: Java 21 Record 클래스나 Spring Controller 코드를 분석하여 MCP 표준 JSON Schema(Tool 명세)를 자동 생성합니다.
|
||||
scope: workspace
|
||||
---
|
||||
|
||||
# 시스템 지침 (System Prompt)
|
||||
너는 MCP(Model Context Protocol) Tool 스키마 전문 엔지니어다.
|
||||
사용자가 Java 코드를 제공하면 다음 규칙에 따라 JSON Schema를 작성해라.
|
||||
|
||||
1. **타입 매핑:** Java 21 `Record`의 필드 타입을 완벽하게 JSON Schema 타입(string, integer, boolean, object, array 등)으로 매핑할 것.
|
||||
2. **필수값 추출:** `@NotNull`, `@NotBlank` 또는 Java 21 Record의 Non-null 필드는 반드시 `required` 배열에 포함시킬 것.
|
||||
3. **설명 추가:** JavaDoc 주석이 있거나 필드 위에 주석이 있다면 스키마의 `description` 필드로 적극 반영할 것.
|
||||
|
||||
# 출력 형식
|
||||
결과물은 반드시 순수한 JSON 형태로만 출력하고, 불필요한 부연 설명은 생략한다.
|
||||
249
.agents/skills/mcp_best_practices/SKILL.md
Normal file
249
.agents/skills/mcp_best_practices/SKILL.md
Normal file
@@ -0,0 +1,249 @@
|
||||
# MCP Server Best Practices
|
||||
|
||||
## Quick Reference
|
||||
|
||||
### Server Naming
|
||||
- **Python**: `{service}_mcp` (e.g., `slack_mcp`)
|
||||
- **Node/TypeScript**: `{service}-mcp-server` (e.g., `slack-mcp-server`)
|
||||
|
||||
### Tool Naming
|
||||
- Use snake_case with service prefix
|
||||
- Format: `{service}_{action}_{resource}`
|
||||
- Example: `slack_send_message`, `github_create_issue`
|
||||
|
||||
### Response Formats
|
||||
- Support both JSON and Markdown formats
|
||||
- JSON for programmatic processing
|
||||
- Markdown for human readability
|
||||
|
||||
### Pagination
|
||||
- Always respect `limit` parameter
|
||||
- Return `has_more`, `next_offset`, `total_count`
|
||||
- Default to 20-50 items
|
||||
|
||||
### Transport
|
||||
- **Streamable HTTP**: For remote servers, multi-client scenarios
|
||||
- **stdio**: For local integrations, command-line tools
|
||||
- Avoid SSE (deprecated in favor of streamable HTTP)
|
||||
|
||||
---
|
||||
|
||||
## Server Naming Conventions
|
||||
|
||||
Follow these standardized naming patterns:
|
||||
|
||||
**Python**: Use format `{service}_mcp` (lowercase with underscores)
|
||||
- Examples: `slack_mcp`, `github_mcp`, `jira_mcp`
|
||||
|
||||
**Node/TypeScript**: Use format `{service}-mcp-server` (lowercase with hyphens)
|
||||
- Examples: `slack-mcp-server`, `github-mcp-server`, `jira-mcp-server`
|
||||
|
||||
The name should be general, descriptive of the service being integrated, easy to infer from the task description, and without version numbers.
|
||||
|
||||
---
|
||||
|
||||
## Tool Naming and Design
|
||||
|
||||
### Tool Naming
|
||||
|
||||
1. **Use snake_case**: `search_users`, `create_project`, `get_channel_info`
|
||||
2. **Include service prefix**: Anticipate that your MCP server may be used alongside other MCP servers
|
||||
- Use `slack_send_message` instead of just `send_message`
|
||||
- Use `github_create_issue` instead of just `create_issue`
|
||||
3. **Be action-oriented**: Start with verbs (get, list, search, create, etc.)
|
||||
4. **Be specific**: Avoid generic names that could conflict with other servers
|
||||
|
||||
### Tool Design
|
||||
|
||||
- Tool descriptions must narrowly and unambiguously describe functionality
|
||||
- Descriptions must precisely match actual functionality
|
||||
- Provide tool annotations (readOnlyHint, destructiveHint, idempotentHint, openWorldHint)
|
||||
- Keep tool operations focused and atomic
|
||||
|
||||
---
|
||||
|
||||
## Response Formats
|
||||
|
||||
All tools that return data should support multiple formats:
|
||||
|
||||
### JSON Format (`response_format="json"`)
|
||||
- Machine-readable structured data
|
||||
- Include all available fields and metadata
|
||||
- Consistent field names and types
|
||||
- Use for programmatic processing
|
||||
|
||||
### Markdown Format (`response_format="markdown"`, typically default)
|
||||
- Human-readable formatted text
|
||||
- Use headers, lists, and formatting for clarity
|
||||
- Convert timestamps to human-readable format
|
||||
- Show display names with IDs in parentheses
|
||||
- Omit verbose metadata
|
||||
|
||||
---
|
||||
|
||||
## Pagination
|
||||
|
||||
For tools that list resources:
|
||||
|
||||
- **Always respect the `limit` parameter**
|
||||
- **Implement pagination**: Use `offset` or cursor-based pagination
|
||||
- **Return pagination metadata**: Include `has_more`, `next_offset`/`next_cursor`, `total_count`
|
||||
- **Never load all results into memory**: Especially important for large datasets
|
||||
- **Default to reasonable limits**: 20-50 items is typical
|
||||
|
||||
Example pagination response:
|
||||
```json
|
||||
{
|
||||
"total": 150,
|
||||
"count": 20,
|
||||
"offset": 0,
|
||||
"items": [...],
|
||||
"has_more": true,
|
||||
"next_offset": 20
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Transport Options
|
||||
|
||||
### Streamable HTTP
|
||||
|
||||
**Best for**: Remote servers, web services, multi-client scenarios
|
||||
|
||||
**Characteristics**:
|
||||
- Bidirectional communication over HTTP
|
||||
- Supports multiple simultaneous clients
|
||||
- Can be deployed as a web service
|
||||
- Enables server-to-client notifications
|
||||
|
||||
**Use when**:
|
||||
- Serving multiple clients simultaneously
|
||||
- Deploying as a cloud service
|
||||
- Integration with web applications
|
||||
|
||||
### stdio
|
||||
|
||||
**Best for**: Local integrations, command-line tools
|
||||
|
||||
**Characteristics**:
|
||||
- Standard input/output stream communication
|
||||
- Simple setup, no network configuration needed
|
||||
- Runs as a subprocess of the client
|
||||
|
||||
**Use when**:
|
||||
- Building tools for local development environments
|
||||
- Integrating with desktop applications
|
||||
- Single-user, single-session scenarios
|
||||
|
||||
**Note**: stdio servers should NOT log to stdout (use stderr for logging)
|
||||
|
||||
### Transport Selection
|
||||
|
||||
| Criterion | stdio | Streamable HTTP |
|
||||
|-----------|-------|-----------------|
|
||||
| **Deployment** | Local | Remote |
|
||||
| **Clients** | Single | Multiple |
|
||||
| **Complexity** | Low | Medium |
|
||||
| **Real-time** | No | Yes |
|
||||
|
||||
---
|
||||
|
||||
## Security Best Practices
|
||||
|
||||
### Authentication and Authorization
|
||||
|
||||
**OAuth 2.1**:
|
||||
- Use secure OAuth 2.1 with certificates from recognized authorities
|
||||
- Validate access tokens before processing requests
|
||||
- Only accept tokens specifically intended for your server
|
||||
|
||||
**API Keys**:
|
||||
- Store API keys in environment variables, never in code
|
||||
- Validate keys on server startup
|
||||
- Provide clear error messages when authentication fails
|
||||
|
||||
### Input Validation
|
||||
|
||||
- Sanitize file paths to prevent directory traversal
|
||||
- Validate URLs and external identifiers
|
||||
- Check parameter sizes and ranges
|
||||
- Prevent command injection in system calls
|
||||
- Use schema validation (Pydantic/Zod) for all inputs
|
||||
|
||||
### Error Handling
|
||||
|
||||
- Don't expose internal errors to clients
|
||||
- Log security-relevant errors server-side
|
||||
- Provide helpful but not revealing error messages
|
||||
- Clean up resources after errors
|
||||
|
||||
### DNS Rebinding Protection
|
||||
|
||||
For streamable HTTP servers running locally:
|
||||
- Enable DNS rebinding protection
|
||||
- Validate the `Origin` header on all incoming connections
|
||||
- Bind to `127.0.0.1` rather than `0.0.0.0`
|
||||
|
||||
---
|
||||
|
||||
## Tool Annotations
|
||||
|
||||
Provide annotations to help clients understand tool behavior:
|
||||
|
||||
| Annotation | Type | Default | Description |
|
||||
|-----------|------|---------|-------------|
|
||||
| `readOnlyHint` | boolean | false | Tool does not modify its environment |
|
||||
| `destructiveHint` | boolean | true | Tool may perform destructive updates |
|
||||
| `idempotentHint` | boolean | false | Repeated calls with same args have no additional effect |
|
||||
| `openWorldHint` | boolean | true | Tool interacts with external entities |
|
||||
|
||||
**Important**: Annotations are hints, not security guarantees. Clients should not make security-critical decisions based solely on annotations.
|
||||
|
||||
---
|
||||
|
||||
## Error Handling
|
||||
|
||||
- Use standard JSON-RPC error codes
|
||||
- Report tool errors within result objects (not protocol-level errors)
|
||||
- Provide helpful, specific error messages with suggested next steps
|
||||
- Don't expose internal implementation details
|
||||
- Clean up resources properly on errors
|
||||
|
||||
Example error handling:
|
||||
```typescript
|
||||
try {
|
||||
const result = performOperation();
|
||||
return { content: [{ type: "text", text: result }] };
|
||||
} catch (error) {
|
||||
return {
|
||||
isError: true,
|
||||
content: [{
|
||||
type: "text",
|
||||
text: `Error: ${error.message}. Try using filter='active_only' to reduce results.`
|
||||
}]
|
||||
};
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Testing Requirements
|
||||
|
||||
Comprehensive testing should cover:
|
||||
|
||||
- **Functional testing**: Verify correct execution with valid/invalid inputs
|
||||
- **Integration testing**: Test interaction with external systems
|
||||
- **Security testing**: Validate auth, input sanitization, rate limiting
|
||||
- **Performance testing**: Check behavior under load, timeouts
|
||||
- **Error handling**: Ensure proper error reporting and cleanup
|
||||
|
||||
---
|
||||
|
||||
## Documentation Requirements
|
||||
|
||||
- Provide clear documentation of all tools and capabilities
|
||||
- Include working examples (at least 3 per major feature)
|
||||
- Document security considerations
|
||||
- Specify required permissions and access levels
|
||||
- Document rate limits and performance characteristics
|
||||
602
.agents/skills/mcp_evaluation/SKILL.md
Normal file
602
.agents/skills/mcp_evaluation/SKILL.md
Normal file
@@ -0,0 +1,602 @@
|
||||
# MCP Server Evaluation Guide
|
||||
|
||||
## Overview
|
||||
|
||||
This document provides guidance on creating comprehensive evaluations for MCP servers. Evaluations test whether LLMs can effectively use your MCP server to answer realistic, complex questions using only the tools provided.
|
||||
|
||||
---
|
||||
|
||||
## Quick Reference
|
||||
|
||||
### Evaluation Requirements
|
||||
- Create 10 human-readable questions
|
||||
- Questions must be READ-ONLY, INDEPENDENT, NON-DESTRUCTIVE
|
||||
- Each question requires multiple tool calls (potentially dozens)
|
||||
- Answers must be single, verifiable values
|
||||
- Answers must be STABLE (won't change over time)
|
||||
|
||||
### Output Format
|
||||
```xml
|
||||
<evaluation>
|
||||
<qa_pair>
|
||||
<question>Your question here</question>
|
||||
<answer>Single verifiable answer</answer>
|
||||
</qa_pair>
|
||||
</evaluation>
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Purpose of Evaluations
|
||||
|
||||
The measure of quality of an MCP server is NOT how well or comprehensively the server implements tools, but how well these implementations (input/output schemas, docstrings/descriptions, functionality) enable LLMs with no other context and access ONLY to the MCP servers to answer realistic and difficult questions.
|
||||
|
||||
## Evaluation Overview
|
||||
|
||||
Create 10 human-readable questions requiring ONLY READ-ONLY, INDEPENDENT, NON-DESTRUCTIVE, and IDEMPOTENT operations to answer. Each question should be:
|
||||
- Realistic
|
||||
- Clear and concise
|
||||
- Unambiguous
|
||||
- Complex, requiring potentially dozens of tool calls or steps
|
||||
- Answerable with a single, verifiable value that you identify in advance
|
||||
|
||||
## Question Guidelines
|
||||
|
||||
### Core Requirements
|
||||
|
||||
1. **Questions MUST be independent**
|
||||
- Each question should NOT depend on the answer to any other question
|
||||
- Should not assume prior write operations from processing another question
|
||||
|
||||
2. **Questions MUST require ONLY NON-DESTRUCTIVE AND IDEMPOTENT tool use**
|
||||
- Should not instruct or require modifying state to arrive at the correct answer
|
||||
|
||||
3. **Questions must be REALISTIC, CLEAR, CONCISE, and COMPLEX**
|
||||
- Must require another LLM to use multiple (potentially dozens of) tools or steps to answer
|
||||
|
||||
### Complexity and Depth
|
||||
|
||||
4. **Questions must require deep exploration**
|
||||
- Consider multi-hop questions requiring multiple sub-questions and sequential tool calls
|
||||
- Each step should benefit from information found in previous questions
|
||||
|
||||
5. **Questions may require extensive paging**
|
||||
- May need paging through multiple pages of results
|
||||
- May require querying old data (1-2 years out-of-date) to find niche information
|
||||
- The questions must be DIFFICULT
|
||||
|
||||
6. **Questions must require deep understanding**
|
||||
- Rather than surface-level knowledge
|
||||
- May pose complex ideas as True/False questions requiring evidence
|
||||
- May use multiple-choice format where LLM must search different hypotheses
|
||||
|
||||
7. **Questions must not be solvable with straightforward keyword search**
|
||||
- Do not include specific keywords from the target content
|
||||
- Use synonyms, related concepts, or paraphrases
|
||||
- Require multiple searches, analyzing multiple related items, extracting context, then deriving the answer
|
||||
|
||||
### Tool Testing
|
||||
|
||||
8. **Questions should stress-test tool return values**
|
||||
- May elicit tools returning large JSON objects or lists, overwhelming the LLM
|
||||
- Should require understanding multiple modalities of data:
|
||||
- IDs and names
|
||||
- Timestamps and datetimes (months, days, years, seconds)
|
||||
- File IDs, names, extensions, and mimetypes
|
||||
- URLs, GIDs, etc.
|
||||
- Should probe the tool's ability to return all useful forms of data
|
||||
|
||||
9. **Questions should MOSTLY reflect real human use cases**
|
||||
- The kinds of information retrieval tasks that HUMANS assisted by an LLM would care about
|
||||
|
||||
10. **Questions may require dozens of tool calls**
|
||||
- This challenges LLMs with limited context
|
||||
- Encourages MCP server tools to reduce information returned
|
||||
|
||||
11. **Include ambiguous questions**
|
||||
- May be ambiguous OR require difficult decisions on which tools to call
|
||||
- Force the LLM to potentially make mistakes or misinterpret
|
||||
- Ensure that despite AMBIGUITY, there is STILL A SINGLE VERIFIABLE ANSWER
|
||||
|
||||
### Stability
|
||||
|
||||
12. **Questions must be designed so the answer DOES NOT CHANGE**
|
||||
- Do not ask questions that rely on "current state" which is dynamic
|
||||
- For example, do not count:
|
||||
- Number of reactions to a post
|
||||
- Number of replies to a thread
|
||||
- Number of members in a channel
|
||||
|
||||
13. **DO NOT let the MCP server RESTRICT the kinds of questions you create**
|
||||
- Create challenging and complex questions
|
||||
- Some may not be solvable with the available MCP server tools
|
||||
- Questions may require specific output formats (datetime vs. epoch time, JSON vs. MARKDOWN)
|
||||
- Questions may require dozens of tool calls to complete
|
||||
|
||||
## Answer Guidelines
|
||||
|
||||
### Verification
|
||||
|
||||
1. **Answers must be VERIFIABLE via direct string comparison**
|
||||
- If the answer can be re-written in many formats, clearly specify the output format in the QUESTION
|
||||
- Examples: "Use YYYY/MM/DD.", "Respond True or False.", "Answer A, B, C, or D and nothing else."
|
||||
- Answer should be a single VERIFIABLE value such as:
|
||||
- User ID, user name, display name, first name, last name
|
||||
- Channel ID, channel name
|
||||
- Message ID, string
|
||||
- URL, title
|
||||
- Numerical quantity
|
||||
- Timestamp, datetime
|
||||
- Boolean (for True/False questions)
|
||||
- Email address, phone number
|
||||
- File ID, file name, file extension
|
||||
- Multiple choice answer
|
||||
- Answers must not require special formatting or complex, structured output
|
||||
- Answer will be verified using DIRECT STRING COMPARISON
|
||||
|
||||
### Readability
|
||||
|
||||
2. **Answers should generally prefer HUMAN-READABLE formats**
|
||||
- Examples: names, first name, last name, datetime, file name, message string, URL, yes/no, true/false, a/b/c/d
|
||||
- Rather than opaque IDs (though IDs are acceptable)
|
||||
- The VAST MAJORITY of answers should be human-readable
|
||||
|
||||
### Stability
|
||||
|
||||
3. **Answers must be STABLE/STATIONARY**
|
||||
- Look at old content (e.g., conversations that have ended, projects that have launched, questions answered)
|
||||
- Create QUESTIONS based on "closed" concepts that will always return the same answer
|
||||
- Questions may ask to consider a fixed time window to insulate from non-stationary answers
|
||||
- Rely on context UNLIKELY to change
|
||||
- Example: if finding a paper name, be SPECIFIC enough so answer is not confused with papers published later
|
||||
|
||||
4. **Answers must be CLEAR and UNAMBIGUOUS**
|
||||
- Questions must be designed so there is a single, clear answer
|
||||
- Answer can be derived from using the MCP server tools
|
||||
|
||||
### Diversity
|
||||
|
||||
5. **Answers must be DIVERSE**
|
||||
- Answer should be a single VERIFIABLE value in diverse modalities and formats
|
||||
- User concept: user ID, user name, display name, first name, last name, email address, phone number
|
||||
- Channel concept: channel ID, channel name, channel topic
|
||||
- Message concept: message ID, message string, timestamp, month, day, year
|
||||
|
||||
6. **Answers must NOT be complex structures**
|
||||
- Not a list of values
|
||||
- Not a complex object
|
||||
- Not a list of IDs or strings
|
||||
- Not natural language text
|
||||
- UNLESS the answer can be straightforwardly verified using DIRECT STRING COMPARISON
|
||||
- And can be realistically reproduced
|
||||
- It should be unlikely that an LLM would return the same list in any other order or format
|
||||
|
||||
## Evaluation Process
|
||||
|
||||
### Step 1: Documentation Inspection
|
||||
|
||||
Read the documentation of the target API to understand:
|
||||
- Available endpoints and functionality
|
||||
- If ambiguity exists, fetch additional information from the web
|
||||
- Parallelize this step AS MUCH AS POSSIBLE
|
||||
- Ensure each subagent is ONLY examining documentation from the file system or on the web
|
||||
|
||||
### Step 2: Tool Inspection
|
||||
|
||||
List the tools available in the MCP server:
|
||||
- Inspect the MCP server directly
|
||||
- Understand input/output schemas, docstrings, and descriptions
|
||||
- WITHOUT calling the tools themselves at this stage
|
||||
|
||||
### Step 3: Developing Understanding
|
||||
|
||||
Repeat steps 1 & 2 until you have a good understanding:
|
||||
- Iterate multiple times
|
||||
- Think about the kinds of tasks you want to create
|
||||
- Refine your understanding
|
||||
- At NO stage should you READ the code of the MCP server implementation itself
|
||||
- Use your intuition and understanding to create reasonable, realistic, but VERY challenging tasks
|
||||
|
||||
### Step 4: Read-Only Content Inspection
|
||||
|
||||
After understanding the API and tools, USE the MCP server tools:
|
||||
- Inspect content using READ-ONLY and NON-DESTRUCTIVE operations ONLY
|
||||
- Goal: identify specific content (e.g., users, channels, messages, projects, tasks) for creating realistic questions
|
||||
- Should NOT call any tools that modify state
|
||||
- Will NOT read the code of the MCP server implementation itself
|
||||
- Parallelize this step with individual sub-agents pursuing independent explorations
|
||||
- Ensure each subagent is only performing READ-ONLY, NON-DESTRUCTIVE, and IDEMPOTENT operations
|
||||
- BE CAREFUL: SOME TOOLS may return LOTS OF DATA which would cause you to run out of CONTEXT
|
||||
- Make INCREMENTAL, SMALL, AND TARGETED tool calls for exploration
|
||||
- In all tool call requests, use the `limit` parameter to limit results (<10)
|
||||
- Use pagination
|
||||
|
||||
### Step 5: Task Generation
|
||||
|
||||
After inspecting the content, create 10 human-readable questions:
|
||||
- An LLM should be able to answer these with the MCP server
|
||||
- Follow all question and answer guidelines above
|
||||
|
||||
## Output Format
|
||||
|
||||
Each QA pair consists of a question and an answer. The output should be an XML file with this structure:
|
||||
|
||||
```xml
|
||||
<evaluation>
|
||||
<qa_pair>
|
||||
<question>Find the project created in Q2 2024 with the highest number of completed tasks. What is the project name?</question>
|
||||
<answer>Website Redesign</answer>
|
||||
</qa_pair>
|
||||
<qa_pair>
|
||||
<question>Search for issues labeled as "bug" that were closed in March 2024. Which user closed the most issues? Provide their username.</question>
|
||||
<answer>sarah_dev</answer>
|
||||
</qa_pair>
|
||||
<qa_pair>
|
||||
<question>Look for pull requests that modified files in the /api directory and were merged between January 1 and January 31, 2024. How many different contributors worked on these PRs?</question>
|
||||
<answer>7</answer>
|
||||
</qa_pair>
|
||||
<qa_pair>
|
||||
<question>Find the repository with the most stars that was created before 2023. What is the repository name?</question>
|
||||
<answer>data-pipeline</answer>
|
||||
</qa_pair>
|
||||
</evaluation>
|
||||
```
|
||||
|
||||
## Evaluation Examples
|
||||
|
||||
### Good Questions
|
||||
|
||||
**Example 1: Multi-hop question requiring deep exploration (GitHub MCP)**
|
||||
```xml
|
||||
<qa_pair>
|
||||
<question>Find the repository that was archived in Q3 2023 and had previously been the most forked project in the organization. What was the primary programming language used in that repository?</question>
|
||||
<answer>Python</answer>
|
||||
</qa_pair>
|
||||
```
|
||||
|
||||
This question is good because:
|
||||
- Requires multiple searches to find archived repositories
|
||||
- Needs to identify which had the most forks before archival
|
||||
- Requires examining repository details for the language
|
||||
- Answer is a simple, verifiable value
|
||||
- Based on historical (closed) data that won't change
|
||||
|
||||
**Example 2: Requires understanding context without keyword matching (Project Management MCP)**
|
||||
```xml
|
||||
<qa_pair>
|
||||
<question>Locate the initiative focused on improving customer onboarding that was completed in late 2023. The project lead created a retrospective document after completion. What was the lead's role title at that time?</question>
|
||||
<answer>Product Manager</answer>
|
||||
</qa_pair>
|
||||
```
|
||||
|
||||
This question is good because:
|
||||
- Doesn't use specific project name ("initiative focused on improving customer onboarding")
|
||||
- Requires finding completed projects from specific timeframe
|
||||
- Needs to identify the project lead and their role
|
||||
- Requires understanding context from retrospective documents
|
||||
- Answer is human-readable and stable
|
||||
- Based on completed work (won't change)
|
||||
|
||||
**Example 3: Complex aggregation requiring multiple steps (Issue Tracker MCP)**
|
||||
```xml
|
||||
<qa_pair>
|
||||
<question>Among all bugs reported in January 2024 that were marked as critical priority, which assignee resolved the highest percentage of their assigned bugs within 48 hours? Provide the assignee's username.</question>
|
||||
<answer>alex_eng</answer>
|
||||
</qa_pair>
|
||||
```
|
||||
|
||||
This question is good because:
|
||||
- Requires filtering bugs by date, priority, and status
|
||||
- Needs to group by assignee and calculate resolution rates
|
||||
- Requires understanding timestamps to determine 48-hour windows
|
||||
- Tests pagination (potentially many bugs to process)
|
||||
- Answer is a single username
|
||||
- Based on historical data from specific time period
|
||||
|
||||
**Example 4: Requires synthesis across multiple data types (CRM MCP)**
|
||||
```xml
|
||||
<qa_pair>
|
||||
<question>Find the account that upgraded from the Starter to Enterprise plan in Q4 2023 and had the highest annual contract value. What industry does this account operate in?</question>
|
||||
<answer>Healthcare</answer>
|
||||
</qa_pair>
|
||||
```
|
||||
|
||||
This question is good because:
|
||||
- Requires understanding subscription tier changes
|
||||
- Needs to identify upgrade events in specific timeframe
|
||||
- Requires comparing contract values
|
||||
- Must access account industry information
|
||||
- Answer is simple and verifiable
|
||||
- Based on completed historical transactions
|
||||
|
||||
### Poor Questions
|
||||
|
||||
**Example 1: Answer changes over time**
|
||||
```xml
|
||||
<qa_pair>
|
||||
<question>How many open issues are currently assigned to the engineering team?</question>
|
||||
<answer>47</answer>
|
||||
</qa_pair>
|
||||
```
|
||||
|
||||
This question is poor because:
|
||||
- The answer will change as issues are created, closed, or reassigned
|
||||
- Not based on stable/stationary data
|
||||
- Relies on "current state" which is dynamic
|
||||
|
||||
**Example 2: Too easy with keyword search**
|
||||
```xml
|
||||
<qa_pair>
|
||||
<question>Find the pull request with title "Add authentication feature" and tell me who created it.</question>
|
||||
<answer>developer123</answer>
|
||||
</qa_pair>
|
||||
```
|
||||
|
||||
This question is poor because:
|
||||
- Can be solved with a straightforward keyword search for exact title
|
||||
- Doesn't require deep exploration or understanding
|
||||
- No synthesis or analysis needed
|
||||
|
||||
**Example 3: Ambiguous answer format**
|
||||
```xml
|
||||
<qa_pair>
|
||||
<question>List all the repositories that have Python as their primary language.</question>
|
||||
<answer>repo1, repo2, repo3, data-pipeline, ml-tools</answer>
|
||||
</qa_pair>
|
||||
```
|
||||
|
||||
This question is poor because:
|
||||
- Answer is a list that could be returned in any order
|
||||
- Difficult to verify with direct string comparison
|
||||
- LLM might format differently (JSON array, comma-separated, newline-separated)
|
||||
- Better to ask for a specific aggregate (count) or superlative (most stars)
|
||||
|
||||
## Verification Process
|
||||
|
||||
After creating evaluations:
|
||||
|
||||
1. **Examine the XML file** to understand the schema
|
||||
2. **Load each task instruction** and in parallel using the MCP server and tools, identify the correct answer by attempting to solve the task YOURSELF
|
||||
3. **Flag any operations** that require WRITE or DESTRUCTIVE operations
|
||||
4. **Accumulate all CORRECT answers** and replace any incorrect answers in the document
|
||||
5. **Remove any `<qa_pair>`** that require WRITE or DESTRUCTIVE operations
|
||||
|
||||
Remember to parallelize solving tasks to avoid running out of context, then accumulate all answers and make changes to the file at the end.
|
||||
|
||||
## Tips for Creating Quality Evaluations
|
||||
|
||||
1. **Think Hard and Plan Ahead** before generating tasks
|
||||
2. **Parallelize Where Opportunity Arises** to speed up the process and manage context
|
||||
3. **Focus on Realistic Use Cases** that humans would actually want to accomplish
|
||||
4. **Create Challenging Questions** that test the limits of the MCP server's capabilities
|
||||
5. **Ensure Stability** by using historical data and closed concepts
|
||||
6. **Verify Answers** by solving the questions yourself using the MCP server tools
|
||||
7. **Iterate and Refine** based on what you learn during the process
|
||||
|
||||
---
|
||||
|
||||
# Running Evaluations
|
||||
|
||||
After creating your evaluation file, you can use the provided evaluation harness to test your MCP server.
|
||||
|
||||
## Setup
|
||||
|
||||
1. **Install Dependencies**
|
||||
|
||||
```bash
|
||||
pip install -r scripts/requirements.txt
|
||||
```
|
||||
|
||||
Or install manually:
|
||||
```bash
|
||||
pip install anthropic mcp
|
||||
```
|
||||
|
||||
2. **Set API Key**
|
||||
|
||||
```bash
|
||||
export ANTHROPIC_API_KEY=your_api_key_here
|
||||
```
|
||||
|
||||
## Evaluation File Format
|
||||
|
||||
Evaluation files use XML format with `<qa_pair>` elements:
|
||||
|
||||
```xml
|
||||
<evaluation>
|
||||
<qa_pair>
|
||||
<question>Find the project created in Q2 2024 with the highest number of completed tasks. What is the project name?</question>
|
||||
<answer>Website Redesign</answer>
|
||||
</qa_pair>
|
||||
<qa_pair>
|
||||
<question>Search for issues labeled as "bug" that were closed in March 2024. Which user closed the most issues? Provide their username.</question>
|
||||
<answer>sarah_dev</answer>
|
||||
</qa_pair>
|
||||
</evaluation>
|
||||
```
|
||||
|
||||
## Running Evaluations
|
||||
|
||||
The evaluation script (`scripts/evaluation.py`) supports three transport types:
|
||||
|
||||
**Important:**
|
||||
- **stdio transport**: The evaluation script automatically launches and manages the MCP server process for you. Do not run the server manually.
|
||||
- **sse/http transports**: You must start the MCP server separately before running the evaluation. The script connects to the already-running server at the specified URL.
|
||||
|
||||
### 1. Local STDIO Server
|
||||
|
||||
For locally-run MCP servers (script launches the server automatically):
|
||||
|
||||
```bash
|
||||
python scripts/evaluation.py \
|
||||
-t stdio \
|
||||
-c python \
|
||||
-a my_mcp_server.py \
|
||||
evaluation.xml
|
||||
```
|
||||
|
||||
With environment variables:
|
||||
```bash
|
||||
python scripts/evaluation.py \
|
||||
-t stdio \
|
||||
-c python \
|
||||
-a my_mcp_server.py \
|
||||
-e API_KEY=abc123 \
|
||||
-e DEBUG=true \
|
||||
evaluation.xml
|
||||
```
|
||||
|
||||
### 2. Server-Sent Events (SSE)
|
||||
|
||||
For SSE-based MCP servers (you must start the server first):
|
||||
|
||||
```bash
|
||||
python scripts/evaluation.py \
|
||||
-t sse \
|
||||
-u https://example.com/mcp \
|
||||
-H "Authorization: Bearer token123" \
|
||||
-H "X-Custom-Header: value" \
|
||||
evaluation.xml
|
||||
```
|
||||
|
||||
### 3. HTTP (Streamable HTTP)
|
||||
|
||||
For HTTP-based MCP servers (you must start the server first):
|
||||
|
||||
```bash
|
||||
python scripts/evaluation.py \
|
||||
-t http \
|
||||
-u https://example.com/mcp \
|
||||
-H "Authorization: Bearer token123" \
|
||||
evaluation.xml
|
||||
```
|
||||
|
||||
## Command-Line Options
|
||||
|
||||
```
|
||||
usage: evaluation.py [-h] [-t {stdio,sse,http}] [-m MODEL] [-c COMMAND]
|
||||
[-a ARGS [ARGS ...]] [-e ENV [ENV ...]] [-u URL]
|
||||
[-H HEADERS [HEADERS ...]] [-o OUTPUT]
|
||||
eval_file
|
||||
|
||||
positional arguments:
|
||||
eval_file Path to evaluation XML file
|
||||
|
||||
optional arguments:
|
||||
-h, --help Show help message
|
||||
-t, --transport Transport type: stdio, sse, or http (default: stdio)
|
||||
-m, --model Claude model to use (default: claude-3-7-sonnet-20250219)
|
||||
-o, --output Output file for report (default: print to stdout)
|
||||
|
||||
stdio options:
|
||||
-c, --command Command to run MCP server (e.g., python, node)
|
||||
-a, --args Arguments for the command (e.g., server.py)
|
||||
-e, --env Environment variables in KEY=VALUE format
|
||||
|
||||
sse/http options:
|
||||
-u, --url MCP server URL
|
||||
-H, --header HTTP headers in 'Key: Value' format
|
||||
```
|
||||
|
||||
## Output
|
||||
|
||||
The evaluation script generates a detailed report including:
|
||||
|
||||
- **Summary Statistics**:
|
||||
- Accuracy (correct/total)
|
||||
- Average task duration
|
||||
- Average tool calls per task
|
||||
- Total tool calls
|
||||
|
||||
- **Per-Task Results**:
|
||||
- Prompt and expected response
|
||||
- Actual response from the agent
|
||||
- Whether the answer was correct (✅/❌)
|
||||
- Duration and tool call details
|
||||
- Agent's summary of its approach
|
||||
- Agent's feedback on the tools
|
||||
|
||||
### Save Report to File
|
||||
|
||||
```bash
|
||||
python scripts/evaluation.py \
|
||||
-t stdio \
|
||||
-c python \
|
||||
-a my_server.py \
|
||||
-o evaluation_report.md \
|
||||
evaluation.xml
|
||||
```
|
||||
|
||||
## Complete Example Workflow
|
||||
|
||||
Here's a complete example of creating and running an evaluation:
|
||||
|
||||
1. **Create your evaluation file** (`my_evaluation.xml`):
|
||||
|
||||
```xml
|
||||
<evaluation>
|
||||
<qa_pair>
|
||||
<question>Find the user who created the most issues in January 2024. What is their username?</question>
|
||||
<answer>alice_developer</answer>
|
||||
</qa_pair>
|
||||
<qa_pair>
|
||||
<question>Among all pull requests merged in Q1 2024, which repository had the highest number? Provide the repository name.</question>
|
||||
<answer>backend-api</answer>
|
||||
</qa_pair>
|
||||
<qa_pair>
|
||||
<question>Find the project that was completed in December 2023 and had the longest duration from start to finish. How many days did it take?</question>
|
||||
<answer>127</answer>
|
||||
</qa_pair>
|
||||
</evaluation>
|
||||
```
|
||||
|
||||
2. **Install dependencies**:
|
||||
|
||||
```bash
|
||||
pip install -r scripts/requirements.txt
|
||||
export ANTHROPIC_API_KEY=your_api_key
|
||||
```
|
||||
|
||||
3. **Run evaluation**:
|
||||
|
||||
```bash
|
||||
python scripts/evaluation.py \
|
||||
-t stdio \
|
||||
-c python \
|
||||
-a github_mcp_server.py \
|
||||
-e GITHUB_TOKEN=ghp_xxx \
|
||||
-o github_eval_report.md \
|
||||
my_evaluation.xml
|
||||
```
|
||||
|
||||
4. **Review the report** in `github_eval_report.md` to:
|
||||
- See which questions passed/failed
|
||||
- Read the agent's feedback on your tools
|
||||
- Identify areas for improvement
|
||||
- Iterate on your MCP server design
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Connection Errors
|
||||
|
||||
If you get connection errors:
|
||||
- **STDIO**: Verify the command and arguments are correct
|
||||
- **SSE/HTTP**: Check the URL is accessible and headers are correct
|
||||
- Ensure any required API keys are set in environment variables or headers
|
||||
|
||||
### Low Accuracy
|
||||
|
||||
If many evaluations fail:
|
||||
- Review the agent's feedback for each task
|
||||
- Check if tool descriptions are clear and comprehensive
|
||||
- Verify input parameters are well-documented
|
||||
- Consider whether tools return too much or too little data
|
||||
- Ensure error messages are actionable
|
||||
|
||||
### Timeout Issues
|
||||
|
||||
If tasks are timing out:
|
||||
- Use a more capable model (e.g., `claude-3-7-sonnet-20250219`)
|
||||
- Check if tools are returning too much data
|
||||
- Verify pagination is working correctly
|
||||
- Consider simplifying complex questions
|
||||
37
.gitea/workflows/deploy.yml
Normal file
37
.gitea/workflows/deploy.yml
Normal file
@@ -0,0 +1,37 @@
|
||||
name: Deploy to OCIWP
|
||||
|
||||
on:
|
||||
push:
|
||||
branches:
|
||||
- main
|
||||
- master
|
||||
workflow_dispatch:
|
||||
|
||||
jobs:
|
||||
deploy:
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- name: Deploy Task on Host
|
||||
run: |
|
||||
echo "Starting Standard CI/CD Deploy pipeline..."
|
||||
|
||||
# 1. gradle 공식 이미지로 바로 컴파일 빌드!
|
||||
docker run --rm \
|
||||
-v /home/ubuntu/app/ax_hub_mcp_tool:/app \
|
||||
-v /var/run/docker.sock:/var/run/docker.sock \
|
||||
-v /home/ubuntu/.gradle:/home/gradle/.gradle \
|
||||
-w /app \
|
||||
gradle:8-jdk21 \
|
||||
./gradlew bootJar -x test
|
||||
|
||||
# 2. 빌드 결과 plain jar 아카이브 정리
|
||||
find /app -name "*-plain.jar" -delete
|
||||
|
||||
# 3. 마운트된 /app 디렉토리로 이동하여 호스트의 도커 컴포즈 제어! (러너 자신을 파괴하지 않고 나머지 서비스만 안전하게 재생성)
|
||||
cd /app
|
||||
docker compose up -d --build gateway redis mci-mock dozzle tool-sms tool-email tool-other tool-payment
|
||||
|
||||
# 4. 배포 후 대롱대롱 매달려 있는 가비지 이미지 자동 소거 청소!
|
||||
docker image prune -f
|
||||
|
||||
echo "CI/CD Deploy Success!"
|
||||
62
.gitignore
vendored
62
.gitignore
vendored
@@ -56,11 +56,69 @@ build/
|
||||
# ===== VS Code =====
|
||||
.vscode/
|
||||
|
||||
# ===== 鍮뚮뱶 寃곌낵臾?=====
|
||||
target/
|
||||
!**/src/main/**/target/
|
||||
!**/src/test/**/target/
|
||||
*.class
|
||||
*.jar
|
||||
!gradle/wrapper/gradle-wrapper.jar
|
||||
*.war
|
||||
*.ear
|
||||
*.nar
|
||||
|
||||
# ===== 濡쒓렇 =====
|
||||
*.log
|
||||
logs/
|
||||
spring-shell.log
|
||||
|
||||
# ===== Maven =====
|
||||
.mvn/wrapper/maven-wrapper.jar
|
||||
pom.xml.tag
|
||||
pom.xml.releaseBackup
|
||||
pom.xml.versionsBackup
|
||||
pom.xml.next
|
||||
release.properties
|
||||
dependency-reduced-pom.xml
|
||||
buildNumber.properties
|
||||
.mvn/timing.properties
|
||||
|
||||
# ===== IntelliJ IDEA =====
|
||||
.idea/
|
||||
*.iws
|
||||
*.iml
|
||||
*.ipr
|
||||
out/
|
||||
|
||||
# ===== Eclipse / STS =====
|
||||
.apt_generated
|
||||
.classpath
|
||||
.factorypath
|
||||
.project
|
||||
.settings
|
||||
.springBeans
|
||||
.sts4-cache
|
||||
bin/
|
||||
|
||||
# ===== NetBeans =====
|
||||
/nbproject/private/
|
||||
/nbbuild/
|
||||
/dist/
|
||||
/nbdist/
|
||||
/.nb-gradle/
|
||||
build/
|
||||
.gradle/
|
||||
!**/src/main/**/build/
|
||||
!**/src/test/**/build/
|
||||
|
||||
# ===== VS Code =====
|
||||
.vscode/
|
||||
|
||||
# ===== OS =====
|
||||
.DS_Store
|
||||
Thumbs.db
|
||||
ehthumbs.db
|
||||
|
||||
# ===== ?섍꼍?ㅼ젙 (誘쇨컧?뺣낫 遺꾨━ ?? =====
|
||||
# application-local.properties
|
||||
# application-secret.properties
|
||||
# application-local.yml
|
||||
# application-secret.yml
|
||||
|
||||
16
Dockerfile
16
Dockerfile
@@ -1,30 +1,30 @@
|
||||
# 1. 鍮뚮뱶 ?섍꼍 (JDK 21)
|
||||
# 1. ??š®ë±???<3F>ê¼<C3AA> (JDK 21)
|
||||
FROM eclipse-temurin:21-jdk-alpine AS builder
|
||||
WORKDIR /app
|
||||
|
||||
# Gradle ?섑띁? ?뚯뒪 蹂듭궗
|
||||
# Gradle ??‘ë<E28098><C3AB>?? ???’ª 蹂ë“ê¶?
|
||||
COPY gradlew .
|
||||
COPY gradle gradle
|
||||
COPY build.gradle settings.gradle ./
|
||||
COPY src src
|
||||
|
||||
# 沅뚰븳 遺??諛?鍮뚮뱶 (?뚯뒪???쒖쇅)
|
||||
# æ²…ëš°ë¸??ºÂ€??è«???š®ë±?(???’ª????–쇅)
|
||||
RUN chmod +x gradlew
|
||||
RUN ./gradlew clean build -x test
|
||||
|
||||
# 2. ?ㅽ뻾 ?섍꼍 (JRE 21)
|
||||
# 2. ??½ë»¾ ??<3F>ê¼<C3AA> (JRE 21)
|
||||
FROM eclipse-temurin:21-jre-alpine
|
||||
WORKDIR /app
|
||||
|
||||
# ??꾩〈 ?ㅼ젙 (?쒓뎅 ?쒓컙)
|
||||
# ???꾩ã€???¼ì ™ (??“뎅 ??“ì»™)
|
||||
RUN apk add --no-cache tzdata
|
||||
ENV TZ=Asia/Seoul
|
||||
|
||||
# 鍮뚮뱶??JAR ?뚯씪 蹂듭궗
|
||||
# ??š®ë±??JAR ???”ª 蹂ë“ê¶?
|
||||
COPY --from=builder /app/build/libs/*.jar app.jar
|
||||
|
||||
# 湲곕낯 ?ы듃 ?몄텧
|
||||
# 湲곕???????몄텧
|
||||
EXPOSE 8081
|
||||
|
||||
# 而⑦뀒?대꼫 ?ㅽ뻾 ??JAR ?ㅽ뻾
|
||||
# ?Œâ‘¦???€ê¼???½ë»¾ ??JAR ??½ë»¾
|
||||
ENTRYPOINT ["java", "-jar", "app.jar"]
|
||||
|
||||
@@ -1,3 +1,17 @@
|
||||
/**
|
||||
* @package io.shinhanlife
|
||||
* @className McpBridge
|
||||
* @description AX HUB 시스템 처리 클래스
|
||||
* @author 0986406
|
||||
* @create 2026.09.01
|
||||
* <pre>
|
||||
* ---------- 개정이력 ----------
|
||||
* 수정일 수정자 수정내용
|
||||
* ---------- -------- ---------------------------
|
||||
* 2026.09.01 0986406 최초생성
|
||||
*
|
||||
* </pre>
|
||||
*/
|
||||
import java.io.BufferedReader;
|
||||
import java.io.InputStreamReader;
|
||||
import java.net.URI;
|
||||
|
||||
191
README.md
191
README.md
@@ -1,177 +1,168 @@
|
||||
# AXHUB Backend
|
||||
# DAP Backend
|
||||
|
||||
Spring Boot 湲곕컲 AXHUB 愿由ъ옄 諛깆뿏??API ?쒕쾭 諛?MCP(Model Context Protocol) Gateway / Tool 遺꾩궛 ?쒕쾭 ?꾨줈?앺듃?낅땲??
|
||||
Spring Boot 기반 DAP 관리자 백엔드 API 서버 및 MCP(Model Context Protocol) Gateway / Tool 분산 서버 프로젝트입니다.
|
||||
|
||||
---
|
||||
|
||||
## ?꾪궎?띿쿂 媛쒖슂 (Architecture Overview)
|
||||
## 아키텍처 개요 (Architecture Overview)
|
||||
|
||||
AXHUB Backend???ш쾶 2媛쒖쓽 二쇱슂 ?좏뵆由ъ??댁뀡 怨꾩링?쇰줈 遺꾨━ ?댁쁺?⑸땲??
|
||||
DAP Backend는 2개의 주요 애플리케이션으로 분리 운영됩니다:
|
||||
|
||||
1. **MCP Gateway (`AxHubGatewayApplication`)**: ?댁옣????UI(AI 梨쀫큸, ?ㅼ틦?대뜑) ?쒓났 諛??몃? LLM(Claude, GPT ?? ?쒕쾭??MCP ?듭떊??諛쏆븘 ?대? Tool ?쒕쾭?ㅻ줈 遺꾨같(?쇱슦???섎뒗 ?덈툕 ?쒕쾭 (?ы듃: 8081)
|
||||
2. **MCP Tool (`AxHubToolApplication`)**: ?ㅼ젣 ?덇굅???쒖뒪??MCI, EAI ??怨??듭떊?섏뿬 鍮꾩쫰?덉뒪 濡쒖쭅(寃곗젣, ?닿??좎껌 ?????섑뻾?섎뒗 ?대뙌???쒕쾭 (?ы듃: 8082~8085 ??遺꾩궛 援ъ꽦 媛??
|
||||
1. **MCP Gateway (`DapGatewayApplication`)**: 외부 LLM(Claude, GPT 등) 서버의 MCP 통신을 받아, 내부 Tool 서버들로 분배(라우팅)하는 허브 서버이자 관리자 웹(Scaffolder)을 제공하는 통합 서버 (포트: 8081)
|
||||
2. **MCP Tool (`DapTool*Application`)**: 실제 레거시 시스템(MCI, EAI 등)과 통신하여 비즈니스 로직(결제, 휴가신청 등)을 수행하는 어댑터 서버 (포트: 8082~8085 등 분산 구성 가능)
|
||||
|
||||
---
|
||||
|
||||
## ?섍꼍 (Environment)
|
||||
## 환경 (Environment)
|
||||
|
||||
| ??ぉ | 踰꾩쟾 |
|
||||
| 항목 | 버전 |
|
||||
|------|------|
|
||||
| Java | 21 |
|
||||
| Spring Boot | 4.0.5 |
|
||||
| Build Tool | Gradle |
|
||||
| 二쇱슂 湲곗닠 ?ㅽ깮 | MyBatis, Lombok, MapStruct, P6Spy |
|
||||
| ?곗씠?곕쿋?댁뒪 | H2 (in-memory, 濡쒖뺄 媛쒕컻?? |
|
||||
| ?몄뀡/罹먯떆 ??μ냼 | Redis |
|
||||
| **?μ븷 寃⑸━ / ?쒖뼱** | **Resilience4j (RateLimiter, CircuitBreaker, Retry)** |
|
||||
| **硫붿떆吏 ??* | **Kafka (?몃옒????< ???湲곗뿴 ?꾪솚??** |
|
||||
| 주요 기술 스택 | MyBatis, Lombok, MapStruct, P6Spy |
|
||||
| 데이터베이스 | H2 (in-memory, 로컬 개발용) |
|
||||
| 세션/캐시 저장소 | Redis |
|
||||
| **장애 격리 / 제어** | **Resilience4j (RateLimiter, CircuitBreaker, Retry)** |
|
||||
| **메시지 큐** | **Kafka (트래픽 폭주 시 대기열 전환용)** |
|
||||
|
||||
---
|
||||
|
||||
## ???ㅽ뻾 諛⑸쾿 (How to Run)
|
||||
## ▶ 실행 방법 (How to Run)
|
||||
|
||||
### 1. Docker Compose瑜??댁슜???꾩껜 ?ㅽ뻾 (沅뚯옣)
|
||||
留덉씠?щ줈?쒕퉬???꾪궎?띿쿂 ?뱀꽦???щ윭 媛쒖쓽 Tool Pod???꾩슂?섎?濡? Docker Compose瑜??댁슜???꾩껜 ?ㅽ뻾??沅뚯옣?⑸땲??
|
||||
|
||||
```bash
|
||||
# ?꾩껜 ?쒖뒪??Gateway + Redis + 紐⑤뱺 Tool Pod) 鍮뚮뱶 諛?諛깃렇?쇱슫???ㅽ뻾
|
||||
./gradlew build -x test
|
||||
docker compose up -d --build
|
||||
```
|
||||
- **Gateway (Chat UI & ?쇱슦??**: `http://localhost:8281`
|
||||
- Gateway媛 ?⑤㈃ ?댁옣??梨쀫큸 ??UI(`http://localhost:8281/chat.html`)???묒냽?섏뿬 諛붾줈 ?뚯뒪?명븷 ???덉뒿?덈떎.
|
||||
|
||||
### 2. 濡쒖뺄 媛쒕컻 ??媛쒕퀎 ?ㅽ뻾 (IntelliJ / Gradle)
|
||||
媛쒕컻 以??뱀젙 紐⑤뱢留??꾩썙 ?붾쾭源낇빐????寃쎌슦 ?꾨옒? 媛숈씠 ?ㅽ뻾?⑸땲??
|
||||
|
||||
- **Gateway ?쒕쾭 湲곕룞:**
|
||||
### 1. Gateway & Tool 서버 실행 (MCP 연동용)
|
||||
- **Gateway 서버 기동:**
|
||||
- `./gradlew :dap-gateway:bootRun`
|
||||
- **Tool ?쒕쾭 湲곕룞 (?? other ??:**
|
||||
- `./gradlew :dap-tool-other:bootRun`
|
||||
- Tool ?쒕쾭媛 湲곕룞?섎㈃ ?먮룞?쇰줈 Gateway???먯떊???깅줉(Auto-Registration)?⑸땲??
|
||||
|
||||
- **Tool 서버 기동:**
|
||||
- `./gradlew :dap-tool-other:bootRun` (또는 dap-tool-payment 등)
|
||||
- Tool 서버가 기동되면 자동으로 Gateway(8081)에 자신을 등록(Auto-Registration)합니다.
|
||||
- **(선택) 특정 Tool 그룹만 실행하기:**
|
||||
- 업무 특성에 따라 세분화된 그룹에 속한 Tool만 띄우고 싶다면, 실행 인수에 `--mcp.tool.target=그룹명`을 추가합니다.
|
||||
- **지원되는 그룹명:**
|
||||
- `NOTIFICATION`: 이메일, SMS 발송
|
||||
- `CLAIM`: 청구 처리, 심사 상태 조회
|
||||
- `POLICY`: 증권 발행, 발행 가능 여부 조회
|
||||
- `HR`: 휴가 등록, 연차 갯수 조회
|
||||
- `CONTRACT`: 계약 상태, 계약 상세 조회
|
||||
- `CUSTOMER`: 고객 등급, 고객 상세 정보 조회
|
||||
- `SAMPLE`: 날씨, 환율, 명언 조회 등 외부 연동 샘플
|
||||
- IntelliJ IDEA: `Run/Debug Configurations`에서 `DapTool*Application` 의 `Program arguments` 에 `--mcp.tool.target=NOTIFICATION` 입력
|
||||
|
||||
|
||||
---
|
||||
|
||||
## ?쨼 AI Agent ?곕룞 ?꾪궎?띿쿂 (Spring AI & MCP)
|
||||
## 🤖 AI Agent 연동 아키텍처 (MCP & Agent Builder)
|
||||
|
||||
蹂??쒖뒪?쒖? **?ы듃??Two-Track) AI ?곕룞 ?꾪궎?띿쿂**瑜??쒓났?섏뿬 濡쒖뺄 媛쒕컻 ?섍꼍怨??꾨줈?뺤뀡 ?섍꼍 紐⑤몢瑜??꾨꼍?섍쾶 吏?먰빀?덈떎.
|
||||
본 시스템은 **투트랙(Two-Track) AI 연동 아키텍처**를 제공하여 로컬 개발 환경과 프로덕션 환경 모두를 완벽하게 지원합니다.
|
||||
|
||||
### 1. ?댁옣????梨쀫큸 (Spring AI 湲곕컲) - NEW! ?럦
|
||||
媛??鍮좊Ⅴ怨?吏곴??곸쑝濡?AI ?먯씠?꾪듃瑜??뚯뒪?명븷 ???덈뒗 ?댁옣??梨쀫큸 ?붾㈃???쒓났?⑸땲?? Gateway ?쒕쾭 ?먯껜??**Spring AI (spring-ai-starter-mcp-server-webmvc)** 媛 ?곕룞?섏뼱 ?덉뼱 蹂꾨룄???뚯씠???ㅽ겕由쏀듃???몃? ???놁씠??利됯컖?곸씤 ?뚯뒪?멸? 媛?ν빀?덈떎.
|
||||
### 1. 로컬 코딩 AI (Antigravity, Cursor, Claude Desktop 등) 연동
|
||||
표준 MCP 통신(Stdio)을 요구하는 로컬 AI 에이전트를 위해 자바 기반의 브릿지 스크립트(`McpBridge.java`)를 내장하고 있습니다. 브릿지가 Stdio 요청을 HTTP로 변환하여 로컬 환경의 Gateway(포트: 8281)로 전달합니다.
|
||||
|
||||
- **?묒냽 諛⑸쾿**: Gateway(Docker) 湲곕룞 ??釉뚮씪?곗??먯꽌 `http://localhost:8281/chat.html` ?묒냽
|
||||
- **?숈옉 諛⑹떇**:
|
||||
1. ?ъ슜?먭? 吏덈Ц???낅젰?섎㈃ ?대? `ChatClient` (Gemini API ??濡??꾩넚
|
||||
2. Spring AI媛 ?대? ?덉??ㅽ듃由ъ쓽 Tool 紐⑸줉??遺꾩꽍?섏뿬 ?꾩슂??Tool ?먯깋
|
||||
3. LLM??Tool ?몄텧 ?먮떒 ?? Gateway??`ExecuteService`瑜?嫄곗퀜 Tool Pod??湲곕뒫??吏곸젒 ?ㅽ뻾
|
||||
4. 寃곌낵瑜?LLM???ㅼ떆 ?댁꽍?섏뿬 ?ъ슜?먯뿉寃??먯뿰?대줈 ?묐떟
|
||||
|
||||
### 2. ?꾨줈?뺤뀡 ?대씪?곕뱶 AI (Google Cloud Agent Builder ?? ?곕룞
|
||||
?ㅼ젣 ?쇱씠釉??쒕퉬?ㅼ뿉???숈옉?섎뒗 ?대씪?곕뱶 Agent Builder??REST API 湲곕컲??OpenAPI Spec???붽뎄?⑸땲??
|
||||
`dap-gateway`???대? **Agent Builder 洹쒓꺽??REST API(`/mcp/api/v1/tools/call`)瑜??ㅼ씠?곕툕濡??쒓났**?섎?濡? 蹂꾨룄??釉뚮┸吏???대뙌???놁씠 Endpoint URL怨?Swagger(OpenAPI) 臾몄꽌留??대씪?곕뱶 肄섏넄???깅줉?섎㈃ 利됱떆 ?쇱씠釉?梨쀫큸/?먯씠?꾪듃濡??쒕퉬?ㅽ븷 ???덉뒿?덈떎.
|
||||
|
||||
### 3. (Legacy) 濡쒖뺄 肄붾뵫 AI (Cursor, Claude Desktop ?? ?곕룞
|
||||
?쒖? MCP ?듭떊(Stdio)???붽뎄?섎뒗 濡쒖뺄 AI ?먯씠?꾪듃瑜??꾪빐 ?먮컮 湲곕컲??釉뚮┸吏 ?ㅽ겕由쏀듃(`McpBridge.java`)瑜??댁옣?섍퀬 ?덉뒿?덈떎. 釉뚮┸吏媛 Stdio ?붿껌??HTTP濡?蹂?섑븯??濡쒖뺄 ?섍꼍??Gateway濡??꾨떖?⑸땲??
|
||||
|
||||
- **?ㅼ젙 諛⑸쾿**: IDE??`mcp_config.json` ?ㅼ젙 ?뚯씪???꾨옒? 媛숈씠 ?깅줉?⑸땲??
|
||||
- **설정 방법**: IDE의 `mcp_config.json` 설정 파일에 아래와 같이 등록합니다.
|
||||
```json
|
||||
"mcpServers": {
|
||||
"dap-gateway": {
|
||||
"command": "java",
|
||||
"args": ["C:/?덈?寃쎈줈/dap-backend-main/McpBridge.java"]
|
||||
"args": ["C:/절대경로/dap-backend-main/McpBridge.java"]
|
||||
}
|
||||
}
|
||||
```
|
||||
- **특정 카테고리 툴 필터링**: `McpBridge.java` 내부의 URI 파라미터(`?categoryKey=sample` 등)를 수정하여 원하는 도메인의 툴만 선택적으로 AI에게 학습시킬 수 있습니다.
|
||||
|
||||
### 2. 프로덕션 클라우드 AI (Google Cloud Agent Builder 등) 연동
|
||||
실제 라이브 서비스에서 동작하는 클라우드 Agent Builder는 REST API 기반의 OpenAPI Spec을 요구합니다.
|
||||
`dap-gateway`는 이미 **Agent Builder 규격의 REST API(`/mcp/api/v1/tools/call`)를 네이티브로 제공**하므로, 별도의 브릿지나 어댑터 없이 Endpoint URL과 Swagger(OpenAPI) 문서만 클라우드 콘솔에 등록하면 즉시 라이브 챗봇/에이전트로 서비스할 수 있습니다.
|
||||
|
||||
---
|
||||
|
||||
## 鍮꾧났媛?Tool 愿由?諛?Fallback ?곕룞 (Visibility & Routing)
|
||||
## 비공개 Tool 관리 및 Fallback 연동 (Visibility & Routing)
|
||||
|
||||
????쒖뒪?쒖? MSA 蹂댁븞 諛??꾪궎?띿쿂 ?먯튃???곕씪 Tool??**?덉??ㅽ듃由??깅줉 ?щ?(?쇱슦??**? **API ?몄텧 ?щ?(媛?쒖꽦)**瑜??꾨꼍??遺꾨━?섏뿬 愿由ы빀?덈떎.
|
||||
저희 시스템은 MSA 보안 및 아키텍처 원칙에 따라 Tool의 **레지스트리 등록 여부(라우팅)**와 **API 노출 여부(가시성)**를 완벽히 분리하여 관리합니다.
|
||||
|
||||
1. **`visible = false`**:
|
||||
?덉??ㅽ듃由ъ뿉 ?뺤긽?곸쑝濡??깅줉?섏뼱 寃뚯씠?몄썾?닿? ?숈쟻?쇰줈 ?쇱슦?낇븯吏留? ?대씪?댁뼵?몄뿉寃??쒓났?섎뒗 `/tools/list` API 紐⑸줉?먯꽌???④꺼吏묐땲??
|
||||
레지스트리에 정상적으로 등록되어 게이트웨이가 동적으로 라우팅하지만, 클라이언트에게 제공되는 `/tools/list` API 목록에서는 숨겨집니다.
|
||||
2. **`register = false`**:
|
||||
?대? ?덉??ㅽ듃由?Redis)?????뺣낫瑜??깅줉?섏? ?딆뒿?덈떎 (?몃? ?덉??ㅽ듃由щ? ?낆옄?곸쑝濡??ъ슜??寃쎌슦 ??.
|
||||
??寃쎌슦 寃뚯씠?몄썾?대뒗 `application.properties`??`mcp.gateway.fallback.routes` ?ㅼ젙??李몄“?섏뿬 **Fallback ?뺤쟻 ?쇱슦??*???섑뻾?섎?濡??곕룞??100% 蹂댁옣?⑸땲??
|
||||
내부 레지스트리(Redis)에 툴 정보를 등록하지 않습니다 (외부 레지스트리를 독자적으로 사용할 경우 등).
|
||||
이 경우 게이트웨이는 `application.yml`의 `mcp.gateway.fallback.routes` 설정을 참조하여 **Fallback 정적 라우팅**을 수행하므로 연동이 100% 보장됩니다.
|
||||
|
||||
```java
|
||||
@McpFunction(
|
||||
name = "secret_tool",
|
||||
visible = false, // 紐⑸줉 ?④? ?щ? (湲곕낯媛? true)
|
||||
register = false // ?대? Redis ?깅줉 ?щ? (湲곕낯媛? true)
|
||||
visible = false, // 목록 숨김 여부 (기본값: true)
|
||||
register = false // 내부 Redis 등록 여부 (기본값: true)
|
||||
)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## ?썳截??쒖뒪???덉젙??諛??ㅽ듃?뚰겕 ?쒖뼱 (Resilience & Network)
|
||||
## 🛡️ 시스템 안정성 및 네트워크 제어 (Resilience & Network)
|
||||
|
||||
MSA 諛??몃? ?쒖뒪??MCI) ?곕룞 ?섍꼍???덉젙?깆쓣 ?꾪빐 ?꾨꼍??3-Tier 諛⑹뼱 泥닿퀎瑜?援ъ텞?덉뒿?덈떎.
|
||||
MSA 및 외부 시스템(MCI) 연동 환경의 안정성을 위해 완벽한 3-Tier 방어 체계를 구축했습니다.
|
||||
|
||||
1. **Gateway ?쇱슦??諛⑹뼱 (Timeout & Fallback):**
|
||||
- MCP ?쇱슦??`McpRouterController`) ?⑥뿉 1珥???꾩븘?껋쓣 媛뺤젣 ?곸슜?섏뿬 ?뱀젙 Tool Pod???묐떟 吏?곗씠 ?꾩껜 ?쒖뒪???μ븷濡??댁뼱吏??寃껋쓣 諛⑹??섍퀬 ?좎냽?섍쾶 ?뺤쟻 Fallback ?쇱슦?낆쑝濡??꾪솚?⑸땲??
|
||||
2. **MCI ?ㅽ듃?뚰겕 ?덉젙??(HTTP/1.1 Downgrade):**
|
||||
- 湲곗〈 HTTP/2 ?ъ슜 ???덇굅???쒖뒪???곕룞 以?媛꾪뿉?곸쑝濡?諛쒖깮?섎뜕 `RST_STREAM` ?ㅻ쪟瑜??먯쿇 李⑤떒?섍린 ?꾪빐, MCI ?꾩슜 `HttpEimsSender`?먮뒗 怨좊룄濡?理쒖쟻?붾맂 **HTTP/1.1 ?꾩슜 而ㅻ꽖???(Factory)**??怨좎젙 ?곸슜?섏뼱 ?ㅽ듃?뚰겕 ?⑥젅??諛⑹??⑸땲??
|
||||
3. **Resilience4j 湲곕컲 ?몃옒???쒖뼱:**
|
||||
- **Gateway 怨꾩링 (?숈쟻 諛⑹뼱):** Tool ?깅줉 ???쒖텧??SLA 硫뷀??곗씠?곕? 湲곕컲?쇰줈 ?숈쟻 CircuitBreaker 諛?RateLimiter瑜?媛?숉븯硫? ?쒓퀎移?珥덇낵 ??Kafka ?먮줈 鍮꾨룞湲??꾪솚?⑸땲??
|
||||
- **Tool 怨꾩링 (?뺤쟻 諛⑹뼱):** ?덇굅??而ㅻ꽖???대???`@CircuitBreaker`, `@RateLimiter` ?대끂?뚯씠??湲곕컲???μ븷 ?꾪뙆 李⑤떒 濡쒖쭅??2李⑥쟻?쇰줈 媛?숇맗?덈떎.
|
||||
1. **Gateway 라우팅 방어 (Timeout & Fallback):**
|
||||
- MCP 라우터(`McpRouterController`) 단에 1초 타임아웃을 강제 적용하여 특정 Tool Pod의 응답 지연이 전체 시스템 장애로 이어지는 것을 방지하고 신속하게 정적 Fallback 라우팅으로 전환합니다.
|
||||
2. **MCI 네트워크 안정화 (HTTP/1.1 Downgrade):**
|
||||
- 기존 HTTP/2 사용 시 레거시 시스템 연동 중 간헐적으로 발생하던 `RST_STREAM` 오류를 원천 차단하기 위해, MCI 전용 `HttpEimsSender`에는 고도로 최적화된 **HTTP/1.1 전용 커넥션 풀(Factory)**이 고정 적용되어 네트워크 단절을 방지합니다.
|
||||
3. **Resilience4j 기반 트래픽 제어:**
|
||||
- **Gateway 계층 (동적 방어):** Tool 등록 시 제출된 SLA 메타데이터를 기반으로 동적 CircuitBreaker 및 RateLimiter를 가동하며, 한계치 초과 시 Kafka 큐로 비동기 전환합니다.
|
||||
- **Tool 계층 (정적 방어):** 레거시 커넥터 내부에 `@CircuitBreaker`, `@RateLimiter` 어노테이션 기반의 장애 전파 차단 로직이 2차적으로 가동됩니다.
|
||||
|
||||
---
|
||||
|
||||
## 紐⑤뱢(Pod) 諛?Tool 肄붾뱶 ?먮룞 ?앹꽦 (Scaffolders)
|
||||
## 모듈(Pod) 및 Tool 코드 자동 생성 (Scaffolders)
|
||||
|
||||
?덈줈???꾨찓?몄쓽 湲곕뒫??異붽?????諛쒖깮?섎뒗 諛섎났?곸씤 ?ㅼ젙(蹂댁씪?ы뵆?덉씠?? ?ㅼ젙 ?뚯씪 蹂듭궗 ????1珥?留뚯뿉 ?먮룞?뷀븯湲??꾪빐 **AXHUB Developer Portal (Web UI)** 諛?**CLI ?ㅼ틦?대뜑 2醫?*???쒓났?⑸땲??
|
||||
새로운 도메인의 기능을 추가할 때 발생하는 반복적인 설정(보일러플레이트, 설정 파일 복사 등)을 1초 만에 자동화하기 위해 **DAP Developer Portal (Web UI)** 및 **CLI 스캐폴더 2종**을 제공합니다.
|
||||
|
||||
### 1. AXHUB Developer Portal (Web UI) - 媛??異붿쿇?섎뒗 諛⑹떇!
|
||||
?댁젣 ???댁긽 ?곕??먯뿉??紐낅졊?대? 移??꾩슂媛 ?놁뒿?덈떎. Gateway 紐⑤뱢???댁옣?????붾㈃?먯꽌 鍮덉뭏留?梨꾩슦硫?肄붾뱶媛 留덈쾿泥섎읆 李랁? ?섏샃?덈떎.
|
||||
### 1. DAP Developer Portal (Web UI) - 가장 추천하는 방식!
|
||||
이제 더 이상 터미널에서 명령어를 칠 필요가 없습니다. Gateway 모듈에 내장된 웹 화면에서 빈칸만 채우면 코드가 마법처럼 찍혀 나옵니다.
|
||||
|
||||
1. **?묒냽 諛⑸쾿**: Gateway ?쒕쾭 湲곕룞 ??釉뚮씪?곗??먯꽌 `http://localhost:8081/admin/scaffold.html` ?묒냽
|
||||
2. **Pod (紐⑤뱢) ?앹꽦 ??*: 紐⑤뱢紐??? hr)怨??ы듃留??낅젰?섎㈃ ?낅┰?곸씤 Spring Boot 紐⑤뱢???붾젆?좊━遺??鍮뚮뱶 ?ㅽ겕由쏀듃源뚯? ?꾨꼍???앹꽦?⑸땲??
|
||||
3. **Tool (湲곕뒫) ?앹꽦 ??*: ?앹꽦??紐⑤뱢???덈줈?????쒕퉬??DTO) 肄붾뱶瑜??먮룞?쇰줈 二쇱엯?⑸땲??
|
||||
1. **접속 방법**: Gateway 서버 기동 후 브라우저에서 `http://localhost:8081/admin/scaffold.html` 접속
|
||||
2. **Pod (모듈) 생성 탭**: 모듈명(예: hr)과 포트만 입력하면 독립적인 Spring Boot 모듈이 디렉토리부터 빌드 스크립트까지 완벽히 생성됩니다.
|
||||
3. **Tool (기능) 생성 탭**: 생성된 모듈에 새로운 툴(서비스/DTO) 코드를 자동으로 주입합니다.
|
||||
|
||||
### 2. CLI ?ㅼ틦?대뜑 (湲곗〈 ?곕???諛⑹떇)
|
||||
???붾㈃???ъ슜?????녿뒗 ?섍꼍?닿굅???곕??먯씠 ?듭닕??寃쎌슦, ?꾨옒 紐낅졊?대? ?듯빐 CLI 留덈쾿?щ? ?ъ슜?????덉뒿?덈떎.
|
||||
### 2. CLI 스캐폴더 (기존 터미널 방식)
|
||||
웹 화면을 사용할 수 없는 환경이거나 터미널이 익숙한 경우, 아래 명령어를 통해 CLI 마법사를 사용할 수 있습니다.
|
||||
|
||||
### 1???덈줈??Pod(紐⑤뱢) ?꾩껜瑜??앹꽦???? `PodScaffolder`
|
||||
?덈줈???꾨찓???? 寃곗젣, HR)???꾪븳 ?꾩쟾???낅┰?곸씤 Spring Boot 紐⑤뱢???앹꽦?⑸땲?? ?대뜑 援ъ“, 鍮뚮뱶 ?ㅽ겕由쏀듃, 媛곸쥌 ?꾨줈?쇳떚 諛??꾩빱 ?ㅼ젙源뚯? ?꾨꼍?섍쾶 ?명똿?⑸땲??
|
||||
### 1⃣ 새로운 Pod(모듈) 전체를 생성할 때: `PodScaffolder`
|
||||
새로운 도메인(예: 결제, HR)을 위한 완전히 독립적인 Spring Boot 모듈을 생성합니다. 폴더 구조, 빌드 스크립트, 각종 프로퍼티 및 도커 설정까지 완벽하게 세팅됩니다.
|
||||
|
||||
```bash
|
||||
# ?ъ슜踰? javac濡?而댄뙆?????ㅽ뻾
|
||||
javac -encoding UTF-8 dap-common/src/main/java/io/shinhanlife/axhub/common/util/PodScaffolder.java
|
||||
java -cp dap-common/src/main/java io.shinhanlife.dap.common.util.PodScaffolder [紐⑤뱢紐? [?ы듃踰덊샇]
|
||||
# 사용법: javac로 컴파일 후 실행
|
||||
javac -encoding UTF-8 dap-common/src/main/java/io/shinhanlife/dap/common/util/PodScaffolder.java
|
||||
java -cp dap-common/src/main/java io.shinhanlife.dap.common.util.PodScaffolder [모듈명] [포트번호]
|
||||
|
||||
# ?ㅽ뻾 ?덉떆 (dap-tool-hr 紐⑤뱢??8086 ?ы듃濡??앹꽦)
|
||||
# 실행 예시 (dap-tool-hr 모듈을 8086 포트로 생성)
|
||||
java -cp dap-common/src/main/java io.shinhanlife.dap.common.util.PodScaffolder hr 8086
|
||||
```
|
||||
|
||||
### 2???앹꽦??紐⑤뱢???덈줈????Function)??異붽????? `ToolScaffolder`
|
||||
?대끂?뚯씠??`@McpTool`, `@McpFunction`)???꾨꼍???щ┛ Service? ?낆텧??DTO 肄붾뱶瑜?吏?뺣맂 紐⑤뱢 ?⑦궎吏 猷곗뿉 留욎떠 ?먮룞 ?앹꽦?⑸땲??
|
||||
### 2⃣ 생성된 모듈에 새로운 툴(Function)을 추가할 때: `ToolScaffolder`
|
||||
어노테이션(`@McpTool`, `@McpFunction`)이 완벽히 달린 Service와 입출력 DTO 코드를 지정된 모듈 패키지 룰에 맞춰 자동 생성합니다.
|
||||
|
||||
```bash
|
||||
# ?ъ슜踰? javac濡?而댄뙆?????ㅽ뻾
|
||||
javac -encoding UTF-8 dap-common/src/main/java/io/shinhanlife/axhub/common/util/ToolScaffolder.java
|
||||
java -cp dap-common/src/main/java io.shinhanlife.dap.common.util.ToolScaffolder [Tool?대쫫] [?명꽣?섏씠?짪D] "[湲곕뒫?ㅻ챸]" "[洹몃9紐?" "[?듭떊諛⑹떇]" "[紐⑤뱢紐?"
|
||||
# 사용법: javac로 컴파일 후 실행
|
||||
javac -encoding UTF-8 dap-common/src/main/java/io/shinhanlife/dap/common/util/ToolScaffolder.java
|
||||
java -cp dap-common/src/main/java io.shinhanlife.dap.common.util.ToolScaffolder [Tool이름] [인터페이스ID] "[기능설명]" "[그룹명]" "[통신방식]" "[모듈명]"
|
||||
|
||||
# ?ㅽ뻾 ?덉떆 (payment 紐⑤뱢??寃곗젣 ?뱀씤 湲곕뒫 異붽?)
|
||||
java -cp dap-common/src/main/java io.shinhanlife.dap.common.util.ToolScaffolder PaymentApproval PAY_001 "寃곗젣 ?뱀씤 泥섎━ 湲곕뒫" "COMMON" "HTTP" "dap-tool-payment"
|
||||
# 실행 예시 (payment 모듈에 결제 승인 기능 추가)
|
||||
java -cp dap-common/src/main/java io.shinhanlife.dap.common.util.ToolScaffolder PaymentApproval PAY_001 "결제 승인 처리 기능" "COMMON" "HTTP" "dap-tool-payment"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## ?⑦궎吏 援ъ“ (Package Structure)
|
||||
## 패키지 구조 (Package Structure)
|
||||
|
||||
```text
|
||||
dap-backend-main (Root)
|
||||
?쒋?? dap-gateway # MCP ?쇱슦???덈툕 ?쒕쾭 (?몃? LLM怨??듭떊 諛?Tool 遺꾨같)
|
||||
?쒋?? dap-common # 怨듯넻 紐⑤뱢 (Security, Session, Config ??
|
||||
?쒋?? dap-tool-core # Tool 怨듯넻 湲곕뒫 (AbstractMcpToolService, Annotation, Scaffolder)
|
||||
?쒋?? dap-tool-email # [Tool] ?대찓??諛쒖넚 ?뱁솕 ?대뙌??紐⑤뱢
|
||||
?쒋?? dap-tool-sms # [Tool] SMS 諛쒖넚 ?뱁솕 ?대뙌??紐⑤뱢
|
||||
?쒋?? dap-tool-payment # [Tool] 寃곗젣 鍮꾩쫰?덉뒪 ?대뙌??紐⑤뱢 (Scaffolded)
|
||||
?붴?? dap-tool-other # [Tool] 湲고? 鍮꾩쫰?덉뒪(泥?뎄, 怨꾩빟, 怨좉컼, HR ?? ?대뙌??紐⑤뱢
|
||||
├── dap-gateway # MCP 라우팅 허브 서버 (외부 LLM과 통신 및 Tool 분배)
|
||||
├── dap-common # 공통 모듈 (Security, Session, Config 등)
|
||||
├── dap-tool-core # Tool 공통 기능 (AbstractMcpToolService, Annotation, Scaffolder)
|
||||
├── dap-tool-email # [Tool] 이메일 발송 특화 어댑터 모듈
|
||||
├── dap-tool-sms # [Tool] SMS 발송 특화 어댑터 모듈
|
||||
├── dap-tool-payment # [Tool] 결제 비즈니스 어댑터 모듈 (Scaffolded)
|
||||
└── dap-tool-other # [Tool] 기타 비즈니스(청구, 계약, 고객, HR 등) 어댑터 모듈
|
||||
```
|
||||
|
||||
*(李멸퀬: 湲곗〈 ?⑥씪 紐⑤뱢 ?꾨줈?앺듃?먯꽌 留덉씠?щ줈?쒕퉬???뺤옣???꾪빐 紐⑤뱢蹂꾨줈 遺꾨━?섏뿀?쇰ʼn, 媛?Tool ?쒕쾭???낅┰?곸쑝濡??뺤옣 諛?諛고룷?????덉뒿?덈떎.)*
|
||||
*(참고: 기존 단일 모듈 프로젝트에서 마이크로서비스 확장을 위해 모듈별로 분리되었으며, 각 Tool 서버는 독립적으로 확장 및 배포할 수 있습니다.)*
|
||||
# | ||||