> ## Documentation Index
> Fetch the complete documentation index at: https://docs.runlayer.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Troubleshooting

> Common issues and solutions for AI Watch package, Detect, Enforce, and Sessions deployments

## Authentication and Configuration

Package-based AI Watch uses one organization API key for scans, hooks, sessions, and macOS updates. The key is written by the macOS Configuration Profile (`OrgApiKey`) or Windows MSI property (`AIWATCH_ORG_API_KEY`) and read by the installed `aiwatch` binary.

| HTTP Status | Error                      | Cause                                                                                                                                                  | Resolution                                                                                                                                                                                                                                                            |
| ----------- | -------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| 401         | Missing or invalid API key | The key is missing from managed config, malformed, or has been revoked/rotated                                                                         | Verify `OrgApiKey` (macOS) / `OrgApiKey` (Windows) is present and current in the AI Watch configuration; if it was rotated, regenerate the configuration from **Settings → MDM Configuration** and re-push                                                            |
| 403         | Valid key, wrong scope     | The org API key authenticated but lacks the **Shadow AI Scan** (`mcp_watch_scan`) role, or is being used outside the AI Watch and self-update surfaces | Use a key minted with the **Shadow AI Scan** role. The AI Watch key is intentionally down-scoped to submission plus narrow update-target and installer-download endpoints; it cannot read tenant telemetry, organization settings, rollout policy, or release history |
| 500         | Server error               | Internal server issue                                                                                                                                  | Contact Runlayer support                                                                                                                                                                                                                                              |

Manual `runlayer scan` workflows can still use interactive login credentials or a stored organization API key. Legacy enrollment keys only apply to older non-package auto-provisioning flows.

The AI Watch organization API key is tenant-scoped: it is not tied to a user, cannot impersonate one, and is blocked from policy-based access control. On the backend it is stored only as an HMAC-SHA256 hash — the plaintext is shown once at creation and cannot be retrieved later, so if it is lost, regenerate the configuration to mint a new one.

## Interpreting Discovery Results

<AccordionGroup>
  <Accordion title="User appears in the Shadow report for MCP servers they say they never used">
    **Cause:** Detect classifies based on **MCP config files found on disk**, not actual tool calls. A user being attributed to a shadow MCP means a matching config was found on a device whose login username resolved to that user — not that they ever invoked any tool from the server.

    Common reasons a config gets attributed to a user who never used it:

    * **Project-level config in a cloned repo.** A `.cursor/mcp.json`, `.vscode/mcp.json`, `.mcp.json`, `.codex/config.toml`, or similar checked into a shared repo is discovered on every developer who clones it. The config is attributed to the OS user who ran the scan on that machine.
    * **Stale or experimental config.** The user added the server to a client config file once (e.g. trying it out, copy-pasting from a blog), never invoked it, and forgot it was still there.
    * **Shared device.** Another OS account on the same workstation configured the MCP, but the scan ran under a different OS account that resolves to this Runlayer user.
    * **Username resolution mismatch.** Tiered matching can link a device's OS username to the wrong Runlayer user. See [User and Identity Mapping](/shadow-ai/identity-mapping), then fix via **Settings → MDM Configuration → Unresolved usernames** and [re-analyze](/shadow-ai/re-analyzing).

    **How the user (or admin) can verify the source themselves:**

    1. Click the shadow artifact row in the **Shadow** page to open the detail drawer.
    2. The detail includes the **device hostname**, **device id**, the MCP server's `command` / `args` / `url`, and any **project paths** where the config was found.
    3. If a project path is listed, the discovery came from a project-level config file. Open that repo and inspect the checked-in MCP config to confirm.
    4. If no project path is listed, the discovery came from a global client config. On the listed device, inspect the relevant client config (e.g. `~/Library/Application Support/Claude/claude_desktop_config.json`, `~/.cursor/mcp.json`, `~/.codex/config.toml`).
    5. To clear the discovery, remove the entry from the offending config file (or uninstall the project) and the next scan will drop it from the inventory.
  </Accordion>

  <Accordion title="A skill is flagged High or Medium risk but is legitimate">
    **Cause:** Detect assigns each discovered skill a risk level (High, Medium, Low, Minimal) by analyzing the skill's instructions for risky patterns — prompt-injection phrasing, data-exfiltration, or unsafe-automation language. The risk level is a heuristic signal for triage, **not** a verdict that the skill is malicious. A legitimate internal skill can score High or Medium if its instructions resemble those patterns (for example, a sanctioned skill that reads files and posts to an API).

    **What to do:**

    1. Open the skill's row in **Shadow → Skills** to read the analyzed instructions and see why it was flagged.
    2. If the skill is sanctioned, bring it under management by publishing it through Runlayer — managed skills are classified **Managed** instead of **Shadow**. See [Responding to Discoveries](/shadow-ai/responding-to-discoveries).
    3. After the skill's content or management status changes, [re-analyze](/shadow-ai/re-analyzing) to refresh its classification and risk level.

    Detect does not block skills — risk levels are for inventory and review only. Blocking applies to shadow MCP sources and local tool calls via [Enforce](/shadow-ai/enforce).
  </Accordion>
</AccordionGroup>

## Package Issues

<AccordionGroup>
  <Accordion title="No devices appearing in the Shadow dashboard">
    **Quick fixes:**

    1. Confirm the package is installed:
       ```bash theme={null}
       test -x /usr/local/bin/aiwatch && echo "macOS aiwatch installed"
       ```
       ```powershell theme={null}
       Test-Path "C:\Program Files\Runlayer\AIWatch\aiwatch.exe"
       ```
    2. Confirm managed configuration is present:
       ```bash theme={null}
       defaults read /Library/Managed\ Preferences/com.runlayer.aiwatch.plist
       ```
       ```powershell theme={null}
       Get-ItemProperty "HKLM:\Software\Runlayer\AIWatch"
       ```
    3. Trigger one scan manually from the device:
       ```bash theme={null}
       /usr/local/bin/aiwatch scan
       ```
       ```powershell theme={null}
       & "C:\Program Files\Runlayer\AIWatch\aiwatch.exe" scan
       ```
    4. Confirm network connectivity from the device to your Runlayer host.
  </Accordion>

  <Accordion title="macOS scan agent is not running">
    **Quick fixes:**

    1. Confirm the bundled LaunchAgent exists:
       ```bash theme={null}
       test -f /Library/LaunchAgents/com.runlayer.aiwatch.plist
       ```
    2. Check whether launchd loaded it for the logged-in user:
       ```bash theme={null}
       launchctl print "gui/$(id -u)/com.runlayer.aiwatch"
       ```
    3. Kickstart it once:
       ```bash theme={null}
       launchctl kickstart -k "gui/$(id -u)/com.runlayer.aiwatch"
       ```
    4. Inspect the unified log:
       ```bash theme={null}
       log show --predicate 'process == "aiwatch"' --last 1h
       ```
  </Accordion>

  <Accordion title="`aiwatch: host not configured` on macOS">
    **Cause:** The AI Watch Configuration Profile has not synced yet, or it did not write `Host` / `OrgApiKey` into managed preferences.

    **Resolution:**

    1. Check managed preferences:
       ```bash theme={null}
       defaults read /Library/Managed\ Preferences/com.runlayer.aiwatch.plist
       ```
    2. If the file is empty or missing, force an MDM sync from the device or wait for the next check-in.
    3. Confirm the profile targets the same device group as the `.pkg`.
  </Accordion>

  <Accordion title={`log show --predicate 'process == "aiwatch"' returns nothing`}>
    **Cause:** The LaunchAgent has not loaded, no scan has run yet, or unified log history has already rolled off.

    **Resolution:**

    1. Check whether launchd loaded the agent for the logged-in user:
       ```bash theme={null}
       launchctl print "gui/$(id -u)/com.runlayer.aiwatch"
       ```
    2. If launchd returns `Could not find service`, re-deploy the `.pkg`; the postinstall may not have bootstrapped the agent.
    3. If the agent is loaded, wait one `StartInterval` tick (default 15 minutes) or seed output with:
       ```bash theme={null}
       /usr/local/bin/aiwatch scan
       ```
    4. For durable logs, collect unified logs with `log collect` or forward them to your SIEM. macOS retains unified logs for roughly 24 hours to a few days depending on device log volume.
  </Accordion>

  <Accordion title="Users see 'Background Item Added' notification on first login after install">
    **Cause:** The Login Items profile has not applied yet, or the device is running macOS older than 13 where the payload is ignored harmlessly.

    **Resolution:**

    1. Confirm `com.runlayer.aiwatch.loginitems.mobileconfig` is deployed to the device.
    2. After MDM sync, verify Background Task Management state:
       ```bash theme={null}
       sfltool dumpbtm | grep -i runlayer
       ```
    3. Expect the agent to show `Disposition: enabled allowed visible`.
  </Accordion>

  <Accordion title="macOS package blocked by Gatekeeper">
    **Cause:** The `.pkg` is not signed, not notarized, or is not the official CI-built release artifact.

    **Resolution:**

    1. Verify the package signature and notarization:
       ```bash theme={null}
       pkgutil --check-signature /path/to/aiwatch-<version>-macos-arm64.pkg
       spctl --assess --type install -vv /path/to/aiwatch-<version>-macos-arm64.pkg
       ```
    2. Expected output includes `status: signed by a developer certificate issued by Apple` and `source=Notarized Developer ID`.
    3. Official GitHub Release builds are signed and notarized by Anysource Inc. with Developer ID team `AF2M8HC7A2`.
  </Accordion>

  <Accordion title="Windows scans are not running">
    Scans run from device-local Scheduled Tasks the MSI registers — there are no Intune Remediations to check.

    **Quick fixes:**

    1. Confirm the app installed `aiwatch.exe` under `C:\Program Files\Runlayer\AIWatch\`.
    2. Confirm the hidden `\Runlayer` task folder and the `AIWatchScan` task exist:
       ```powershell theme={null}
       Get-ScheduledTask -TaskPath "\Runlayer\" | Format-Table TaskName, State
       ```
       The task-registration action gates on the `OrgApiKey` registry value — if there are no tasks, confirm `Get-ItemProperty "HKLM:\Software\Runlayer\AIWatch"` shows `OrgApiKey`, then repair/reinstall the app.
    3. Check the `AIWatchScan` task's last result and the per-profile scan outcomes. The single SYSTEM task runs `aiwatch.exe scan --all-users` and scans each real profile in a child process; a never-signed-in user has no profile yet and is picked up on the next tick (logon / boot / 15 min):
       ```powershell theme={null}
       Get-ScheduledTaskInfo -TaskPath "\Runlayer\" -TaskName "AIWatchScan" | Select-Object LastRunTime, LastTaskResult
       # Orchestrator summary (all_users_scan_complete) + per-profile all_users_profile_scan_* lines:
       Get-Content "C:\ProgramData\Runlayer\Logs\scheduled-task.log" -Tail 40
       # Detailed per-profile child-scan output lives in each scanned user's profile:
       #   C:\Users\<username>\.runlayer\logs
       ```
    4. Check the task-registration lines (tagged `[register]`) in `C:\ProgramData\Runlayer\Logs\scheduled-task.log`.
    5. Run a one-off scan as the logged-in user:
       ```powershell theme={null}
       & "C:\Program Files\Runlayer\AIWatch\aiwatch.exe" scan
       ```
  </Accordion>

  <Accordion title="Hook configs are not installed or drifted">
    **Quick fixes:**

    1. Confirm Enforce or Sessions is enabled. If both are disabled, AI Watch is Detect-only and no hooks are installed.
    2. Check hook config compliance:
       ```bash theme={null}
       /usr/local/bin/aiwatch setup hooks check --mdm
       ```
       ```powershell theme={null}
       & "C:\Program Files\Runlayer\AIWatch\aiwatch.exe" setup hooks check --mdm
       ```
    3. On macOS, confirm the bootstrap daemon is loaded:
       ```bash theme={null}
       sudo launchctl print system/com.runlayer.aiwatch.bootstrap
       ```
    4. On Windows, confirm the device-local `AIWatchHooks` task (SYSTEM) is running: `Get-ScheduledTaskInfo -TaskPath "\Runlayer\" -TaskName "AIWatchHooks" | Select-Object LastRunTime, LastTaskResult`. Force a run with `Start-ScheduledTask -TaskPath "\Runlayer\" -TaskName "AIWatchHooks"`, or repair/reinstall the app if it's missing.
    5. Restart Cursor, VS Code, Claude Code, Codex, or Hermes after hook configs change.
  </Accordion>

  <Accordion title="Hooks are installed but not blocking">
    **Quick fixes:**

    1. Confirm `Enforcement` is enabled:
       ```bash theme={null}
       defaults read /Library/Managed\ Preferences/com.runlayer.aiwatch.plist
       ```
       ```powershell theme={null}
       Get-ItemProperty "HKLM:\Software\Runlayer\AIWatch"
       ```
    2. `Enforcement=false`, `AIWATCH_ENFORCEMENT=0`, or an absent enforcement key is monitoring-only. Hooks may still forward Sessions telemetry, but they do not block.
    3. Re-push the configuration with `Enforcement=true` / `AIWATCH_ENFORCEMENT=1`.
    4. Restart the AI client so it reloads hook config.
  </Accordion>

  <Accordion title="Scan or hook fails behind a corporate TLS inspection proxy">
    **Symptoms:** The scan discovers local config, then fails while submitting results:

    ```text theme={null}
    Submitting 9 servers from 9 configs (1 global, 8 project)...
    Scan failed error='[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1017)' error_type=ConnectError
    ```

    **Cause:** Some enterprise proxies intercept HTTPS and re-sign traffic with a private root CA. The Runlayer CLI and packaged `aiwatch` binary trust the OS trust store automatically, but scans and hooks can fail if that private CA is not installed in system trust.

    **Resolution:**

    1. Prefer installing the corporate root CA into the OS trust store through MDM.
    2. For debugging or one-off deployments, pass a PEM bundle explicitly:
       ```bash theme={null}
       runlayer --ca-bundle /path/to/corporate-ca.pem scan
       ```
    3. For custom CLI workflows, set `RUNLAYER_CA_BUNDLE`. The CLI also honors `SSL_CERT_FILE` and `REQUESTS_CA_BUNDLE` as fallbacks.
    4. Ask the security team to exempt the Runlayer tenant hostname from TLS inspection, such as `*.runlayer.com` or customer-managed hostnames like `runlayer.dev.company.com` just like they do for other security products like endpoint detection and response (EDR).
  </Accordion>

  <Accordion title="runlayer: command not found after install">
    **Cause:** `uv tool install runlayer` places the binary in `~/.local/bin` (macOS/Linux) or `%USERPROFILE%\.local\bin` (Windows), which may not be in your shell's PATH yet.

    **Resolution:**

    1. Restart your terminal (new shells pick up the updated PATH)
    2. Or run `uv tool update-shell` to add `~/.local/bin` to your shell profile, then `exec $SHELL -l` (or open a new terminal)
    3. Verify: `which runlayer` (macOS/Linux) or `where runlayer` (Windows)
  </Accordion>

  <Accordion title="Users see macOS consent dialogs for AI Watch">
    **Symptoms:** Users are prompted with macOS permission dialogs asking to grant access to `aiwatch` for directories like Desktop, Documents, or Downloads.

    **Cause:** The AI Watch PPPC profile was not deployed before the package ran, so macOS falls back to interactive consent prompts. This can also happen if a device is not enrolled via User-Approved MDM or Automated Device Enrollment.

    **Resolution:**

    1. Deploy `com.runlayer.aiwatch.pppc.mobileconfig` via your MDM.
    2. Re-deploying the PPPC profile restores access even if a user clicked "Don't Allow".
    3. Ensure devices are enrolled via User-Approved MDM or Automated Device Enrollment (required for MDM-managed PPPC profiles to take effect)
    4. Verify the profile is installed:
       ```bash theme={null}
       profiles show -type configuration | grep -i aiwatch
       ```
    5. Re-run the scan after the profile is applied
  </Accordion>

  <Accordion title="Scan finds fewer MCP servers than expected on macOS">
    **Symptoms:** Scan completes successfully but doesn't find MCP servers you know exist in directories like Desktop, Documents, or Application Support.

    **Cause:** macOS TCC (Transparency, Consent, and Control) is blocking access to protected directories.

    **Resolution:**

    1. Deploy the AI Watch PPPC profile to grant Full Disk Access to `/usr/local/lib/runlayer/aiwatch/aiwatch`.
    2. Ensure devices are enrolled via User-Approved MDM or Automated Device Enrollment
    3. Verify the profile is installed: `profiles show -type configuration | grep aiwatch`
    4. Check TCC denials:
       ```bash theme={null}
       sudo log show --predicate 'subsystem == "com.apple.TCC"' --last 10m | grep aiwatch
       ```
       Look for `Prompting policy ... Deny` lines. TCC PPPC payloads are ignored on manually enrolled MDM devices.
    5. Verify the package binary is installed and signed correctly:
       ```bash theme={null}
       ls -la /usr/local/lib/runlayer/aiwatch/aiwatch
       codesign -dv /usr/local/lib/runlayer/aiwatch/aiwatch
       ```
    6. Re-run the scan after the PPPC profile is applied
  </Accordion>

  <Accordion title="PPPC profile applied but TCC still denies AI Watch">
    **Cause:** The PPPC profile's CodeRequirement does not match the installed binary. The profile pins identifier `com.runlayer.aiwatch` and Developer ID team `AF2M8HC7A2` (Anysource Inc.). Custom, local, or ad-hoc signed builds do not satisfy that requirement, so TCC falls back to deny.

    **Resolution:**

    1. Verify the installed binary:
       ```bash theme={null}
       codesign -dv --verbose=4 /usr/local/lib/runlayer/aiwatch/aiwatch
       ```
    2. Expected output:
       ```text theme={null}
       Authority=Developer ID Application: Anysource Inc. (AF2M8HC7A2)
       Identifier=com.runlayer.aiwatch
       ```
    3. If the team identifier or identifier do not match, confirm with your Runlayer account team that the `.pkg` you deployed is the official signed and notarized build from the GitHub Release.
  </Accordion>

  <Accordion title="Legacy runlayer-scan artifacts still appear">
    **Cause:** The device still has artifacts from the old script-based macOS Detect rollout, such as `/usr/local/bin/runlayer-scan`, `com.runlayer.scan`, or `com.runlayer.scan.pppc`.

    **Resolution:** Follow [Legacy macOS Detect Cleanup](/shadow-ai/deploy/macos-cleanup-script-deployment). For full package removal, use [Remove AI Watch](/shadow-ai/deploy/remove).
  </Accordion>
</AccordionGroup>

## Log Locations and Checks

| Platform            | Feature                         | Log Location                                                                                                                                                                    |
| ------------------- | ------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| macOS package       | Detect / Enforce / Sessions     | Unified log: `log show --predicate 'process == "aiwatch"' --last 1h`                                                                                                            |
| macOS package       | LaunchAgent                     | `launchctl print "gui/$(id -u)/com.runlayer.aiwatch"`                                                                                                                           |
| macOS package       | Bootstrap daemon                | `sudo launchctl print system/com.runlayer.aiwatch.bootstrap`                                                                                                                    |
| macOS package       | Update daemon                   | `sudo launchctl print system/com.runlayer.aiwatch.update` (check state and last exit status); inspect `AutoUpdate` in `/Library/Managed Preferences/com.runlayer.aiwatch.plist` |
| Windows             | Intune                          | `C:\ProgramData\Microsoft\IntuneManagementExtension\Logs`                                                                                                                       |
| Windows package     | Managed config                  | `HKLM\Software\Runlayer\AIWatch`                                                                                                                                                |
| Windows package     | Scan (all-users orchestrator)   | `C:\ProgramData\Runlayer\Logs\scheduled-task.log` (orchestrator summary `all_users_scan_complete` + per-profile `all_users_profile_scan_*` lines)                               |
| Windows package     | Scan (per-profile child output) | `C:\Users\<username>\.runlayer\logs` (each scanned profile; the child scan's `USERPROFILE` points there)                                                                        |
| Manual CLI          | Runlayer CLI                    | Terminal output from `runlayer scan --verbose`                                                                                                                                  |
| Legacy macOS script | Detect                          | `/var/log/runlayer/ai_watch_detect.log`, `/tmp/runlayer-scan.stdout.log`, `/tmp/runlayer-scan.stderr.log`                                                                       |

## Exit Codes

| Code | Meaning                                                                                                                                                                                                                     |
| ---- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| 0    | Success / compliant                                                                                                                                                                                                         |
| 1    | General failure (auth error or unexpected exception) or hook config drift                                                                                                                                                   |
| 2    | `scan`: a backend endpoint returned 404, so findings were not stored (backend too old, wrong `Host`, or the path was intercepted by a proxy/CDN). Other commands: missing input, misconfiguration, or no supported clients. |
| 3    | `scan`: a submission was attempted but failed with a network error or backend 5xx, so some findings were not stored.                                                                                                        |
| 4    | Hook install/check credential gate failed                                                                                                                                                                                   |

A `scan` that finds no MCP servers, skills, or plugins exits `0` (nothing to submit). Servers, skills, and plugins are submitted independently, so a failure in one category does not block the others; if any found item cannot be stored the scan exits nonzero (`2` or `3`). Best-effort check-ins (detect, enforce, sessions) never change the exit code. Any nonzero `scan` exit surfaces in the `AIWatchScan` task's `LastTaskResult` (Windows) / the launchd job log (macOS), so a scan that ran but persisted nothing is not reported as a silent green.

## Verifying PPPC Profile Deployment

After deploying the AI Watch PPPC profile, verify it is installed on target devices:

```bash theme={null}
profiles show -type configuration | grep -i aiwatch

# Check if Full Disk Access is granted (requires admin)
sudo sqlite3 "/Library/Application Support/com.apple.TCC/TCC.db" \
  "SELECT client, auth_value FROM access WHERE service='kTCCServiceSystemPolicyAllFiles';"
```

## Verifying Package Binary

Verify the package binary is properly installed and signed:

```bash theme={null}
# Check package binary is installed
ls -la /usr/local/lib/runlayer/aiwatch/aiwatch

# Verify the signature and identifier
codesign -dv /usr/local/lib/runlayer/aiwatch/aiwatch

# Test a scan
/usr/local/bin/aiwatch scan
```

The signature should show `Identifier=com.runlayer.aiwatch`, which matches the AI Watch PPPC profile.

## Related Resources

<CardGroup cols={2}>
  <Card title="User and Identity Mapping" icon="user" href="/shadow-ai/identity-mapping">
    Resolve device usernames to Runlayer users
  </Card>

  <Card title="FAQ" icon="circle-question" href="/shadow-ai/faq">
    Pre-deployment, security-review, and operational questions
  </Card>

  <Card title="Remove AI Watch" icon="trash" href="/shadow-ai/deploy/remove">
    Remove package-based and legacy script-based deployments
  </Card>
</CardGroup>

## Getting Help

If you continue to experience issues:

1. Collect the relevant log files from the affected device
2. Note the HTTP status codes and error messages
3. Contact Runlayer support with the collected information