Upstream sync

Overview

CyberChef-MCP maintains a fork relationship with GCHQ/CyberChef while removing web UI components and focusing on the MCP server implementation. This guide explains the sync model and automated workflows.

Current state (v2.0.0): upstream base v11.4.0, 504 operations. The sync is a whole-tree rsync -a --delete mirror of src/core/** plus six upstream-owned src/node/*.mjs files, with fork changes carried as patches/fork/*.patch re-applied afterwards, and an allowlist scope check that fails the run on anything out of scope.

The sync is permanently one-way (pull only). As of v2.0.0 the combined work is GPL-3.0-or-later, and GCHQ will not accept GPLv3 into an Apache-2.0 project, so MCP-layer improvements cannot be contributed back. This codifies what was already practice.

Version numbers in the example blocks below (v10.19.4, v10.19.5, 464 operations) are illustrative of the workflow’s output format and are deliberately left as written; they are not a statement of current state.

Architecture

Repository Structure

CyberChef-MCP/
├── src/
│   ├── core/
│   │   └── operations/     # Synced from upstream
│   ├── node/
│   │   └── mcp-server.mjs  # MCP-specific (not synced)
│   └── (no web/ directory - removed)
├── ref-proj/
│   └── CyberChef/          # Full upstream clone (reference)
├── tests/
│   └── mcp/                # MCP tests (not synced)
└── .github/workflows/      # Custom workflows (not synced)

Sync Model

Philosophy: Selective file syncing, NOT git merge

Synced (mirrored verbatim): all of src/core/**, plus the six upstream-owned files in src/node/ — api.mjs, apiUtils.mjs, File.mjs, NodeDish.mjs, NodeRecipe.mjs, repl.mjs.
Never synced (generated): src/core/config/modules/, src/core/config/OperationConfig.json, src/core/operations/index.mjs, src/node/config/OperationConfig.json, src/node/index.mjs. Our .gitignore and upstream’s list exactly these five.
Never synced (fork-owned): everything else in src/node/ — mcp-server.mjs, lib/, transports.mjs, streaming.mjs, recipe-*.mjs, worker*.mjs, errors.mjs, logger.mjs, retry.mjs, deprecation.mjs — plus package.json, Gruntfile.js, tests/mcp/.
Forbidden (must never return): src/web/, tests/browser/, nightwatch.json, postcss.config.js, .devcontainer/. The web app was removed in v1.7.1.
Fork changes to upstream files: kept as patches in patches/fork/ and re-applied after every mirror. See patches/fork/README.md. Never hand-edit an upstream-owned file — the mirror overwrites it and your change disappears silently.
Reference Directory: ref-proj/CyberChef/ is a git submodule holding pristine upstream.

Why the whole tree, not just operations/. The previous model synced only src/core/operations/*.mjs. That cannot express a major-version jump. Measured on 10.19.4 → 11.4.0: 449 files identical, 112 differing, 61 added upstream, 1 removed upstream. The removed file is the clearest illustration: upstream deleted src/core/lib/ImageManipulation.mjs and refactored BlurImage/SharpenImage to use jimp directly. Syncing operations without lib/ orphans the library; syncing lib/ without operations breaks the build. The tree has to move atomically.

Direction: pull only, permanently. As of v2.0.0 this project is distributed under GPL-3.0-or-later while upstream CyberChef is Apache-2.0. GCHQ cannot accept GPLv3 code into an Apache-2.0 project, so changes can no longer be contributed back upstream — the sync is one-way by licence, not merely by convention. See ADR 0001.

A practical consequence: a bug found in a synced upstream operation must be reported to gchq/CyberChef and fixed there. Patching it locally both diverges this fork and is silently reverted by the next sync. Fixes that cannot wait belong in the MCP layer under src/node/, not in src/core/.

Workflows

1. upstream-monitor.yml

Purpose: Detect upstream changes and create tracking issues

Trigger:

Scheduled: Every 6 hours
Manual: gh workflow run upstream-monitor.yml

Process:

Update/clone upstream to ref-proj/CyberChef/
Compare operation files with main codebase
Identify new, modified, and deleted operations
Create issue if changes detected

Output:

GitHub issue with:
- Operation count comparison
- List of new operations (first 20)
- List of modified operations (first 20)
- Link to upstream release notes
- Instructions for triggering sync

Example Issue:

## New Upstream Release Detected

**CyberChef Version**: v10.19.5
**Current Version**: v10.19.4
**Detected**: 2025-12-16 12:00:00 UTC

### Operation Changes
- Current Operations: 464
- Upstream Operations: 465
- New Operations: 1
- Modified Operations: 3

#### New Operations

NewCoolOperation.mjs

#### Modified Operations

AESDecrypt.mjs Base64.mjs SHA3.mjs

2. upstream-sync.yml

Purpose: Selectively sync operation files from upstream

Trigger:

Manual: gh workflow run upstream-sync.yml
Automatic: Add upstream-sync-approved label to monitor issue

Process:

Ensure ref-proj/CyberChef/ is updated to the target upstream tag
Identify changes across the whole synced scope (added, modified, removed)
Mirror the tree with rsync -a --delete, so additions, modifications and deletions apply in one pass
Re-apply patches/fork/*.patch. A patch that no longer applies fails the sync — that is correct behaviour, not a breakage: upstream changed the code it touches and a human must decide whether the fix is still needed
Verify the sync stayed in scope — an allowlist, so anything unexpected fails the run
Regenerate OperationConfig.json
Run comprehensive tests (core + MCP + lint)
Update baseline.json for regression testing
Create PR with detailed changelog

Safety Checks:

No src/web/ directory
No tests/browser/ directory
No nightwatch.json, postcss.config.js, .devcontainer/
MCP-specific files unchanged (package.json, Gruntfile.js, etc.)

Example PR:

## Upstream Sync: v10.19.5

Selective sync from CyberChef v10.19.5 (NOT full merge)

### Changes Summary
- New Operations: 1
- Modified Operations: 3
- Deleted Operations: 0 (kept in MCP)
- Total Operations: 465

### Sync Report
(Detailed file-by-file changes)

### Test Results
✅ All tests and linting passed

### Verification Checklist
- [x] Only operation files synced
- [x] No excluded files
- [x] MCP-specific files preserved
- [ ] Manual testing of new operations

3. rollback.yml

Purpose: Emergency rollback if sync causes issues

Trigger:

Manual only: gh workflow run rollback.yml -f reason="Issue description"

Process:

Capture current state (operation count, ref-proj state)
Rollback to specified commit (or parent commit)
Regenerate configs and baseline
Run tests
Create PR with state comparison

Important: Does NOT automatically rollback ref-proj/CyberChef/

You may need to manually rollback ref-proj if upstream changes caused the issue
PR includes instructions for ref-proj rollback if needed

Example PR:

## Emergency Rollback

**Reason:** Test failures after v10.19.5 sync

### State Comparison
| Metric | Before | After |
|--------|--------|-------|
| Operations | 465 | 464 |
| ref-proj tag | v10.19.5 | v10.19.4 |

### ref-proj/CyberChef State
The ref-proj directory was NOT automatically rolled back.
Current state: v10.19.5

(Instructions for manual rollback if needed)

File Exclusion Rules

Never Sync From Upstream

Web UI (81 files):

src/web/ - All web application code
src/web/static/images/ - Logos, icons, screenshots
src/web/static/fonts/ - Font files

Browser Tests (4 files):

tests/browser/ - Nightwatch.js tests

Config Files (3 files):

nightwatch.json - Browser test configuration
postcss.config.js - CSS processing for web UI
.devcontainer/ - VS Code dev container

Sync Selectively

Primary Sync Target:

src/core/operations/*.mjs - Individual operation implementations

Review Carefully:

src/core/lib/ - Shared library code (may need manual review)
src/core/config/modules/ - Operation module configs

Never Overwrite (MCP-Specific)

Core Files:

package.json - Has mcpVersion field and MCP dependencies
Gruntfile.js - MCP-specific build tasks
src/node/mcp-server.mjs - MCP server entry point
src/node/wrapper.js - MCP wrapper
Dockerfile.mcp - MCP Docker container

Tests & CI:

tests/mcp/ - MCP-specific tests
.github/workflows/ - Custom CI/CD workflows
vitest.config.mjs - Vitest configuration

Documentation:

docs/ - MCP-specific documentation
CLAUDE.md - Project guidance

Common Scenarios

Scenario 1: Routine Upstream Update

Monitor detects new release (automatic every 6 hours)
- Issue created: “New CyberChef release v10.19.5 available”
Review the issue
- Check upstream release notes
- Review operation changes
- Look for breaking changes

Approve sync

gh issue edit <issue-number> --add-label upstream-sync-approved

Sync workflow runs (automatic)
- Selectively copies changed operations
- Runs tests
- Creates PR
Review and merge PR
- Verify test results
- Manual testing if needed
- Merge when ready

Scenario 2: Manual Sync

When to use: Testing, urgent updates, or specific version

# Sync to latest
gh workflow run upstream-sync.yml

# Sync to specific version
gh workflow run upstream-sync.yml -f target_version=v10.19.5

Scenario 3: Sync Causes Issues

Problem: Tests fail or operations broken after sync

# Trigger rollback
gh workflow run rollback.yml -f reason="Test failures after v10.19.5 sync"

Rollback workflow will:

Revert to previous commit
Regenerate configs
Run tests
Create PR for review

If needed, manually rollback ref-proj:

cd ref-proj/CyberChef
git checkout v10.19.4
cd ../..
git add ref-proj/CyberChef
git commit -m "chore: rollback ref-proj to match main codebase"
git push origin rollback-branch

Scenario 4: Upstream Breaking Changes

Problem: Upstream introduces breaking changes we can’t accept

Options:

Skip this release
- Close the monitor issue without syncing
- Document why we’re skipping
- Sync to next stable release
Selective adoption
- Manually cherry-pick specific operations
- Modify operations for MCP compatibility
- Document modifications
Compatibility layer
- Add adapter code in MCP server
- Maintain upstream compatibility
- Test thoroughly

Testing Strategy

Pre-Sync Validation

# Verify ref-proj is clean
cd ref-proj/CyberChef
git status  # Should be clean

# Verify upstream remote
git remote -v  # Should point to gchq/CyberChef

# Check current operation count
ls src/core/operations/*.mjs | wc -l

During Sync

Automated checks in workflow:

Operation count verification
Excluded file detection
MCP-specific file preservation
OperationConfig.json regeneration

Post-Sync Validation

Automated tests:

npm test              # Core operation tests
npm run test:mcp      # MCP validation (343 tests)
npm run lint          # ESLint (zero errors required)

Manual testing:

# Build Docker image
docker build -f Dockerfile.mcp -t cyberchef-mcp .

# Test critical operations
echo '{"jsonrpc":"2.0","id":1,"method":"tools/list"}' | docker run -i --rm cyberchef-mcp
echo '{"jsonrpc":"2.0","id":2,"method":"tools/call","params":{"name":"cyberchef_to_base64","arguments":{"input":"Hello"}}}' | docker run -i --rm cyberchef-mcp

Metrics and Monitoring

Key Metrics

Baseline (v1.7.0):

Operations: 464
MCP Tests: 343
Test Coverage: 78.93% lines, 89.33% functions
Excluded File Violations: 0 (strict requirement)

Success Criteria

For Sync PR Merge:

✅ All operations synced correctly
✅ No excluded files in main codebase
✅ All tests passing (core + MCP)
✅ No MCP-specific files overwritten
✅ ESLint clean (0 errors)
✅ Docker build successful
✅ Manual smoke tests passed

Monitoring

GitHub Actions:

upstream-monitor: Every 6 hours
Check workflow logs for errors
Review created issues

PR Reviews:

Verify sync report accuracy
Check test results
Review changed files list

Troubleshooting

Issue: Monitor creates duplicate issues

Cause: Issue already exists for this version

Solution: Workflow checks for existing issues before creating

Issue: Sync workflow fails with “Excluded files found”

Cause: Upstream added files to excluded paths

Solution:

Review what files were added
Verify they should be excluded
Update exclusion rules if needed
Manual cleanup if necessary

Issue: Tests fail after sync

Cause: Upstream changes incompatible with MCP

Options:

Rollback: Use rollback workflow
Fix Forward: Modify operations for compatibility
Skip Operations: Temporarily exclude problematic operations

Issue: ref-proj directory missing

Cause: First run or directory deleted

Solution: Monitor workflow will clone automatically:

gh workflow run upstream-monitor.yml

Issue: Operation count mismatch

Cause: Manual changes or partial sync

Solution:

Check git status for uncommitted changes
Verify ref-proj is updated
Re-run sync workflow

Best Practices

1. Regular Monitoring

Let scheduled workflow run (every 6 hours)
Review issues promptly
Don’t let sync lag too far behind

2. Careful Review

Always review upstream release notes
Check for breaking changes
Test new operations manually

3. Incremental Sync

Sync one release at a time
Don’t skip multiple versions
Document any manual modifications

4. Test Thoroughly

Run full test suite
Manual testing for critical operations
Docker build verification

5. Document Changes

Update CHANGELOG.md
Note any manual modifications
Document compatibility issues

Workflow Diagrams

Monitor → Sync → Merge Flow

┌─────────────────────┐
│  upstream-monitor   │  (Every 6 hours)
│  - Clone/update ref │
│  - Compare ops      │
│  - Create issue     │
└──────────┬──────────┘
           │
           ▼
┌─────────────────────┐
│   Review Issue      │  (Manual)
│  - Check changes    │
│  - Review notes     │
│  - Add label        │
└──────────┬──────────┘
           │
           │ upstream-sync-approved label
           ▼
┌─────────────────────┐
│   upstream-sync     │  (Automatic)
│  - Selective copy   │
│  - Regen configs    │
│  - Run tests        │
│  - Create PR        │
└──────────┬──────────┘
           │
           ▼
┌─────────────────────┐
│    Review PR        │  (Manual)
│  - Verify changes   │
│  - Test manually    │
│  - Merge            │
└─────────────────────┘

Rollback Flow

┌─────────────────────┐
│   Issue Detected    │
│  - Tests fail       │
│  - Ops broken       │
└──────────┬──────────┘
           │
           ▼
┌─────────────────────┐
│   Trigger Rollback  │  (Manual)
│  gh workflow run    │
│  rollback.yml       │
└──────────┬──────────┘
           │
           ▼
┌─────────────────────┐
│    rollback.yml     │  (Automatic)
│  - Revert commit    │
│  - Regen configs    │
│  - Run tests        │
│  - Create PR        │
└──────────┬──────────┘
           │
           ▼
┌─────────────────────┐
│ Review Rollback PR  │  (Manual)
│  - Verify fix       │
│  - Check ref-proj   │
│  - Merge            │
└─────────────────────┘

References

Upstream Repository: https://github.com/gchq/CyberChef
MCP Server Implementation: src/node/mcp-server.mjs
Workflow Definitions: .github/workflows/upstream-*.yml
Exclusion Rules: See “File Exclusion Rules” section
Architecture Docs: docs/architecture/architecture.md

Version History

v1.7.0 (2025-12-16): Introduced selective sync model
v1.6.2 (2025-12-14): Removed web UI files (88 files)
v1.0.0 (Initial): Full merge model (deprecated)