Labels

Splunk MCP Server Setup and Troubleshooting

Splunk's MCP Server exposes your instance to AI tooling over port 8089. Run SPL, pull metadata, query indexes, interact with the AI Assistant, all through the management API via Model Context Protocol. The app went GA on Feb 4, 2026. The architecture is client-agnostic. Any MCP-capable AI tool works: Claude, Cursor, your own homebrew agent.

Before You Start

Read these first.


Token generation moved. As of v1.0.0, tokens must be generated from within the MCP Server app. Tokens created via Settings > Tokens or the REST API will not work. If you upgraded from the beta, your old tokens are dead. Regenerate from inside the app. The app sets the audience automatically.

The role must be named mcp_user. The app looks for this specific name. An arbitrary role with the right capabilities won't cut it.

Your CLI IP is not your browser IP. If you're in WSL, on a VPN, or behind split tunneling, curl ifconfig.me will return a different IP than your browser. The search-api allow list needs the CLI IP. Miss this and the connection dies silently. No error, no log, nothing.

Windows-native apps need the `wsl` wrapper. Claude Desktop and Cursor run on Windows and can't see `npx` inside your WSL distro without it. If you're already running inside WSL (like Claude Code), skip the wrapper.

Splunk Side

1. Install the Apps

Two apps from Splunkbase:

Without the AI Assistant, those endpoints don't exist. You need both.

2. Enable All MCP Endpoints

After installation, not all MCP endpoints are enabled by default. The app ships with some tools turned off, which means your AI client will only see a partial toolset and silently miss capabilities. Go into the MCP Server app's management UI and verify that every endpoint you need is toggled on.

If you skip this step, everything will appear to work. The proxy connects, the handshake completes, splunk_get_info returns data. But when you try to run SPL or use the AI Assistant tools, they simply won't exist on the client side. No error, no warning, just blank responses from the remote MCP server. The tools aren't advertised because they're disabled server-side. Check this before you burn time debugging a connection that's already working fine.

3. RBAC and Token

Create a role named mcp_user. Assign two capabilities:

  • mcp_tool_execute - grants access to use MCP tools
  • mcp_tool_admin - grants admin access for tool management and token creation

Splunk's setup video says the role doesn't need capabilities. That was true pre-GA. As of v1.0.0, the capabilities are required.

Assign the role to a user, then generate a token from within the MCP Server app. 

4. IP Allow List (Splunk Cloud)

Your client's outbound IP needs to be on the search-api allow list. Port 8089 is the management API where /services/mcp lives. If the IP isn't allowed, the connection dies silently. No error. No log entry. Just the void staring back.

Find your actual outbound IP from wherever your MCP client runs:

curl ifconfig.me

If you're in WSL, on a VPN, behind a proxy, or on a corporate network with split tunneling, this IP is almost certainly different from what your browser reports. Add it via the admin UI or the ACS API:

# Recon: check current allow list
curl https://admin.splunk.com/YOUR-STACK/adminconfig/v2/access/search-api/ipallowlists \
  --header "Authorization: Bearer YOUR_ACS_TOKEN"

# Add your IP
curl -X POST https://admin.splunk.com/YOUR-STACK/adminconfig/v2/access/search-api/ipallowlists \
  --header "Content-Type: application/json" \
  --header "Authorization: Bearer YOUR_ACS_TOKEN" \
  --data '{"subnets": ["YOUR.CLI.IP.ADDRESS/32"]}'

Propagation takes a few minutes. If you're on a VPN with rotating IPs, get the CIDR range from your network team.

Testing Connectivity

Test in layers. Bottom up.

Layer 1: Can You Reach the Endpoint?

Before you involve any tooling, just hit the endpoint with curl. This proves your IP is on the allow list and the network path is clear.

curl -v --connect-timeout 10 \
  https://YOUR-INSTANCE.splunkcloud.com:8089/services/mcp

You're looking for a 200 or 405, and Connection #0 to host ... left intact. Either means the endpoint is alive and your IP is allowed through. If you get a timeout or connection refused, your IP isn't on the search-api allow list. Run curl ifconfig.me, compare it to what's listed, and fix it before going further. Everything else is built on this.

Layer 2: MCP Proxy

Now test the actual MCP transport. This proves the token and protocol handshake:

npx -y mcp-remote \
  https://YOUR-INSTANCE.splunkcloud.com:8089/services/mcp \
  --header "Authorization: Bearer YOUR_TOKEN_HERE"

Success:

Connected to remote server using StreamableHTTPClientTransport
Local STDIO server running
Proxy established successfully between local STDIO and remote StreamableHTTPClientTransport

If Layer 1 passed but this fails, RBAC or your token is the problem. Generated outside the app, or mangled by line breaks.

Client Side Configuration Combos

The connection uses mcp-remote (npm package) to proxy the remote Splunk endpoint to local stdio. Prerequisite: Node.js. Check with which npx. If nothing, sudo apt install nodejs npm.

The Splunk MCP Server app provides a config snippet you can copy directly from its UI. That gets you most of the way. The rest depends on where your client is running.

Claude Desktop (Windows)

Claude Desktop is a Windows-native app. It can't see npx inside WSL without the wsl wrapper. Paste the
config from the MCP app into Claude Desktop's settings, then wrap the command: 

{
    "mcpServers": {
      "splunk-mcp-server": {
        "command": "wsl",
        "args": [
          "npx",
          "-y",
          "mcp-remote",
          "https://YOUR-INSTANCE.splunkcloud.com:8089/services/mcp",
          "--header",
          "Authorization: Bearer YOUR_TOKEN_HERE"
        ]
      }
    }
}

The wsl wrapper calls into your WSL distro where Node.js lives. Without it, Claude Desktop can't find npx and the proxy never starts.


Claude Code (WSL/Linux)

If for some reason you do not want to register the MCP server, maybe you have multiple MCP servers for various projects you can also just drop a .mcp.json in the project root or register it via cli. 


Register it via CLI so it persists across sessions:

claude mcp add splunk-mcp-server -- npx -y mcp-remote \
  https://YOUR-INSTANCE.splunkcloud.com:8089/services/mcp \
  --header "Authorization: Bearer YOUR_TOKEN_HERE"

Or drop a .mcp.json in your project root:

{
    "mcpServers": {
      "splunk-mcp-server": {
        "command": "npx",
        "args": [
          "-y",
          "mcp-remote",
          "https://YOUR-INSTANCE.splunkcloud.com:8089/services/mcp",
          "--header",
          "Authorization: Bearer YOUR_TOKEN_HERE"
        ]
      }
    }
}

Cursor

Same .mcp.json format. If Cursor is running on Windows, use the wsl wrapper like Claude Desktop. If it's running inside WSL, use npx directly.


Test Functionality

From inside a Claude Code session, call splunk_get_info. You should get back your instance version, server name, and health status. If you get data, the channel is open. 


Troubleshooting

Silent connection failure (Splunk Cloud)

IP isn't on the search-api allow list. Run curl ifconfig.me from wherever your MCP client lives, compare it to your browser's IP (they will be different), and add the right one. This is the #1 issue because there's zero feedback when it fails. The void doesn't send error codes.

Warning: ignoring invalid header argument

Your token has line breaks from terminal wrapping. The GA encrypted tokens are long. Copy-paste from a terminal will wrap them across lines and inject invisible characters. Edit the .mcp.json directly in your editor.

insufficient permission to access the resource

Three suspects. Line them up: 1. User has a role named mcp_user with mcp_tool_execute and mcp_tool_admin 2. Token was generated from within the MCP Server app (not Settings > Tokens) 3. You copied the actual token value, not the token ID

500 hostname localhost doesn't match (Enterprise)

The MCP server calls back to itself via REST and can't resolve its own hostname. Set the base URL explicitly:

# $SPLUNK_HOME/etc/apps/Splunk_MCP_Server/local/mcp.conf
[server]
base_url = https://YOUR-SPLUNK-HOST:8089/

self-signed cert in chain (Enterprise)

Default self-signed cert on 8089. Client-side workaround in your .mcp.json:

"env": {
  "NODE_TLS_REJECT_UNAUTHORIZED": "0"
}

Production environments should use a real cert. This is a field expedient, not a permanent solution.

Tools missing or return empty results

v1.0.0 ships with not all endpoints enabled by default. If tools are missing from your client entirely, or you're getting {"results":[],"total_rows":0}, the endpoints are disabled server-side. Open the MCP Server app's management UI and enable the tools you need (see Step 2). This is easy to miss because the connection itself works perfectly. splunk_get_info returns data, the proxy is green, but the AI Assistant tools and others just aren't there.

Quick Reference

| Symptom | Cause | Fix | |---------|-------|-----| | Silent failure, no error | IP not on allow list | curl ifconfig.me then add to search-api allow list | | ignoring invalid header | Token line breaks | Paste in editor, not terminal | | insufficient permission | Token or role issue | Generate token in-app, role named mcp_user with both capabilities | | spawn errors | Missing npx | Install Node.js | | 500 localhost mismatch | No base_url | Set base_url in mcp.conf | | SSL cert errors | Self-signed cert | NODE_TLS_REJECT_UNAUTHORIZED=0 | | Tools missing or empty results | Endpoints not enabled | Enable tools in MCP Server app management UI (Step 2) |

Splunk's troubleshooting video covers the common Enterprise setup failures if you want the visual walkthrough.

Resources

Splunk MCP Server v1.0.0 | Splunk AI Assistant for SPL v1.5.0 | Splunk Cloud v9.3.2411.123 | mcp-remote via npx | WSL2

Labels:

Comments

Post a Comment

Popular Posts