watermarks-remover
About
Strip multi-vendor AI provenance marks: Unicode text hygiene, statistical rewrite hooks, and C2PA/metadata from PNG/JPEG/SVG/PDF/DOCX/HTML/MD
Details
- Author
- guillaumemeyer
- Categories
- Productivity, Other, Design
Jump to
Configuration (env vars for docker compose)
Nothing is required to clean arbitrary text— the core service works out of the box:
echo "Hello\u200bWorld\u00ad!" > /tmp/sample.txt curl -s -X POST http://127.0.0.1:8765/clean -H 'Content-Type: application/json' \ -d "{\"file\": \"$(base64 < /tmp/sample.txt | tr -d '\n')\", \"name\": \"sample.txt\"}"
Everything else is optional and lives in a.envfile at the repo root.docker composeauto-loads.envand interpolates the${VAR}references incompose.yamlfrom it (shell exports win over.envif both are set).
cp .env.example .env # then edit docker compose up -d # picks up .env automatically
.envisgitignored(deny-by-default) — never commit it. For host-side CLI runs (rewrite_text.py, the skill), export the same file into the environment:
set -a; . ./.env; set +a; python3 service/scripts/rewrite_text.py /tmp/x.txt -o /tmp/x.rewritten.txt
Layer B is agent-orchestrated in the skill (it rewrites with its own model), so theWATERMARKS_REWRITE_vars are only needed when drivingrewrite_text.pydirectly.
Images publish automatically onvtags via](#docker--compose).github/workflows/release-images.yml.
inspect_image.pyandclean_image.pycan report a pixel-domain SynthID confidence score when an external checkout of[aloshdenny/reverse-SynthIDis available. The scorer isnot bundled: it is loaded at runtime from your checkout, and its code remains under the upstream project's non-commercial Research License.
_ _ _ ____ ___ ____ ____ _ _ ____ ____ _ _ ____ ____ ____ _ _ ____ _ _ ____ ____ | | | |__| | |___ |__/ |\/| |__| |__/ |_/ [__ __ |__/ |___ |\/| | | | | |___ |__/ |_|_| | | | |___ | \ | | | | | \ | \_ ___] | \ |___ | | |__| \/ |___ | \
Agent skill + stdlib Python service to stripmulti-vendor AI provenance marksfrom text and files — for privacy and hygiene on contentyou own. The skill is a thin client: it drives the machinery over HTTP, so the agent host needs no Python.
Vendors / ecosystems (class-level):Claude,Gemini / SynthID-Text,OpenAIprovenance surfaces,open-LLMKirchenbauer-style (green-list) and keyed-Gumbel / EXP (Aaronson) marks.
Skill path:skills/remove-ai-marks/
Service path:service/
(migration: formerlyremove-claude-marks; slash alias/remove-claude-marksstill documented)
The skill shipsno code— it calls the service over HTTP. Install the skill (markdown only) and start the service, then setWATERMARKS_SERVICE_URLif it is nothttp://127.0.0.1:8765.
# Grok Build / project-local mkdir -p .grok/skills ln -sfn "$(pwd)/skills/remove-ai-marks" .grok/skills/remove-ai-marks # User-global Grok mkdir -p ~/.grok/skills ln -sfn "$(pwd)/skills/remove-ai-marks" ~/.grok/skills/remove-ai-marks
Invoke with/remove-ai-marksor ask to “strip AI watermarks / C2PA / Claude marks / SynthID-class text.”
Install it into~/.cursor/skills/clean-user-facing-text:
On Windows, usepy install_skill.py. Theinstall-skill.shwrapper is provided for macOS/Linux shells. Existing installations are preserved unless you pass--force; replacement is staged first and the previous install is kept as a uniquely named backup.
Skill invocation is model-selected. Projects that explicitly adopt this workflow can also copy the optional rule:
mkdir -p /path/to/project/.cursor/rules cp integrations/cursor/clean-user-facing-text.mdc \ /path/to/project/.cursor/rules/clean-user-facing-text.mdc
For all projects, put the same instruction in CursorUser Rulesinstead. Rules improve consistency but remain model instructions; Cursor does not expose a deterministic pre-send filter for final chat responses.
The fastest path is a local HTTP server (Python 3.10+ stdlib only — no deps, no Docker):
make serve # http://127.0.0.1:8765 # or directly: python3 service/scripts/server.py --host 127.0.0.1 --port 8765
Seedocs/windows-autostart.mdfor auto-starting the service at Windows login without Docker.
For the whole infra (core + optional harness/heavy backends), seeDocker / composebelow.
Optional system tools (auto-used when present — preinstalled in the core Docker image):
Core scripts needPython 3.10+stdlib only. Layer B model calls are optional.
SCRIPTS=service/scripts # Unified inspect / clean python3 "$SCRIPTS/inspect_file.py" draft.md python3 "$SCRIPTS/clean_file.py" draft.md -o draft.cleaned.md python3 "$SCRIPTS/clean_file.py" photo.png -o photo.cleaned.png python3 "$SCRIPTS/clean_file.py" notes.docx -o notes.cleaned.docx # Text Layer A python3 "$SCRIPTS/inspect_text.py" draft.md python3 "$SCRIPTS/clean_text.py" draft.md -o draft.cleaned.md --stats # Layer B rewrite hook (default: print prompt only — no model required) python3 "$SCRIPTS/rewrite_text.py" draft.md --backend print-prompt --strength paraphrase # Optional local Ollama (loopback only by default — remote endpoints require # WATERMARKS_REWRITE_ALLOW_REMOTE=1 or --allow-remote): # WATERMARKS_REWRITE_BACKEND=ollama WATERMARKS_REWRITE_MODEL=llama3.2 \ # python3 "$SCRIPTS/rewrite_text.py" draft.md -o draft.rewritten.md # API keys are read from WATERMARKS_REWRITE_API_KEY only (never argv). # Images python3 "$SCRIPTS/inspect_image.py" shot.png python3 "$SCRIPTS/clean_image.py" shot.png -o shot.cleaned.png
inspect_text.py,clean_text.pyandrewrite_text.pyoperate on text. Pointed at a.docx,.pdfor image they used to decode the compressed bytes and report whatever codepoints fell out — noise that tracks the compression, not the content — andclean_text.pythen wrote those mangled bytes back, destroying the file. They now refuse binary input and name the tool that handles it:
python3 "$SCRIPTS/inspect_text.py" report.docx # refusing to treat report.docx as text: it looks like a ZIP container (DOCX, ODT, …). # Use inspect_file.py / clean_file.py, which route by format, # or pass --force-text to scan the raw bytes anyway.
Detection is by magic number plus a control-byte ratio, so text in encodings other than UTF-8 keeps working.--force-textoverrides it everywhere.
Unrecognized formats are never auto-cleaned
classify()labels bytes that match no supported text, image or container format asunknown— it no longer falls back to "text". In auto modeclean_file.pyrefuses such files (exit 2, no output written) instead of decoding them as UTF-8 and writing back mangled bytes;--as textor--force-textare the explicit opt-ins.inspect_file.pyreports the file asunknown(exit 0), and the HTTP service answers/inspectwithkind: "unknown"but rejects/cleanof unknown formats (400 — send a filename with a known extension, e.g.notes.txt).
The same machinery runs as a stdlib HTTP service (service/scripts/server.py) — the interface the skill uses and the way any web app can integrate without vendoring:
Batch endpoints loop the same per-file pipeline as/inspectand/clean, capped atWATERMARKS_MAX_BATCH_FILESfiles per request (default 50). A malformed entry (bad base64, unknown option, unrecognized format) surfaces as that entry's"ok": falsewith an"error"string — it never aborts the rest of the batch.
WM="http://127.0.0.1:8765" curl -s "$WM/health" # {"ok": true, "version": "..."} curl -s "$WM/openapi.json" # machine-readable OpenAPI 3.0.3 contract curl -s -X POST "$WM/clean" -H 'Content-Type: application/json' \ -d "{\"file\": \"$(base64 < notes.md | tr -d '\n')\", \"name\": \"notes.md\"}"
The service routes by filename extension then magic bytes, so text / image / container are auto-detected. SetWATERMARKS_SERVER_API_KEYto requireAuthorization: Bearer <key>on every request. Loopback-only bind by default (--hostto override); intended for a trusted network.
Watermark detection (/detectanddetect_before/detect_after)
Detection is a separate step from cleaning — the service never calls vendor APIs unless you ask it to:
- POST /detectruns the configured watermark detectors on a file. Text → vendor detectors + stylometry; image → SynthID pixel score.
- /inspectaccepts an opt-in"detect": trueflag that appends detector results to the text report (and can flipsuspicious).
- /cleanaccepts"detect_before"/"detect_after"options to score the input and the cleaned output, so you can measure what a clean actually changed.
Text detectors (see/capabilities→text_detectors):
Image scoring: whenWATERMARKS_SYNTHID_SCORER_URLis set, the service scores images through thewr-synthid-scoresidecar (heavy profile); with a localREVERSE_SYNTHID_DIRit uses the checkout directly. Detection is fail-soft: unconfigured, timed-out, or errored detectors report{"available": false, "error": ...}and never block cleaning.
make docker-core-build docker run --rm -p 127.0.0.1:8765:8765 --read-only --tmpfs /tmp watermarks-remover # any CLI stays runnable by overriding the command: docker run --rm -v "$(pwd):/data" watermarks-remover \ /app/scripts/clean_file.py /data/notes.md -o /data/notes.cleaned.md
docker compose up -d # core HTTP service only docker compose --profile harness up -d # + markllm / markdiffusion docker compose --profile heavy up -d # + ctrlregen / synthid (local builds) docker compose --profile harness --profile heavy up -d # all services
The compose stack maps the core service to127.0.0.1:8765. The harness/heavy services are one-shot CLIs — invoke withdocker compose run --rm <service> …when you need verification or pixel work.
Validate the running stack (exit code only, no output on success):
make compose-check # or: ./compose-check.sh
Checkswr-coreviaGET /healthand runs each harness/heavy service with--help, requiring exit0.
Configuration (env vars for docker compose)
Nothing is required to clean arbitrary text— the core service works out of the box:
echo "Hello\u200bWorld\u00ad!" > /tmp/sample.txt curl -s -X POST http://127.0.0.1:8765/clean -H 'Content-Type: application/json' \ -d "{\"file\": \"$(base64 < /tmp/sample.txt | tr -d '\n')\", \"name\": \"sample.txt\"}"
Everything else is optional and lives in a.envfile at the repo root.docker composeauto-loads.envand interpolates the${VAR}references incompose.yamlfrom it (shell exports win over.envif both are set).
cp .env.example .env # then edit docker compose up -d # picks up .env automatically
.envisgitignored(deny-by-default) — never commit it. For host-side CLI runs (rewrite_text.py, the skill), export the same file into the environment:
set -a; . ./.env; set +a; python3 service/scripts/rewrite_text.py /tmp/x.txt -o /tmp/x.rewritten.txt
Layer B is agent-orchestrated in the skill (it rewrites with its own model), so theWATERMARKS_REWRITE_vars are only needed when drivingrewrite_text.pydirectly.
Images publish automatically onvtags via.github/workflows/release-images.yml.
inspect_image.pyandclean_image.pycan report a pixel-domain SynthID confidence score when an external checkout ofaloshdenny/reverse-SynthIDis available. The scorer isnot bundled: it is loaded at runtime from your checkout, and its code remains under the upstream project's non-commercial Research License.
Option 1: one-command bootstrap (no Docker)
SCRIPTS=service/scripts # Clones upstream, creates a venv, and installs scorer-only dependencies. "$SCRIPTS/setup_synthid.sh" # Score an image (default checkout: ~/reverse-SynthID). REVERSE_SYNTHID_DIR=~/reverse-SynthID \ ~/reverse-SynthID/.venv/bin/python "$SCRIPTS/score_synthid.py" shot.png # Or surface the score from inspect / clean (same venv Python). REVERSE_SYNTHID_DIR=~/reverse-SynthID \ ~/reverse-SynthID/.venv/bin/python "$SCRIPTS/inspect_image.py" shot.png
setup_synthid.shaccepts--dir PATH,--ref REF, and--full(install the full upstreamrequirements.txt, which addstorch/diffusersfor the upstream VAE bypass this project does not use).
On Windows usesetup_synthid.ps1(-Dir,-Ref,-Full), which creates the venv at.venv\Scripts\— the layoutimage_meta.pyalready looks for onos.name == "nt".
make docker-synthid-build # Run unprivileged and with a read-only rootfs; the scorer only needs to read # /data and write to stdout/tmp. docker run --rm \ --user "$(id -u):$(id -g)" \ --read-only --tmpfs /tmp \ -v "$(pwd):/data" \ watermarks-remover-synthid-scorer /data/shot.png
The image is built locally from the upstream source at build time. It is not published, so it does not redistribute the upstream code.
Option 3: HTTP scorer sidecar (docker compose)
Under theheavyprofile the compose stack also runs the scorer as an HTTP sidecar (wr-synthid-score) so thepublished core servicecan score images before/after cleaning without bundling the non-commercial upstream code. Pointwr-coreat it and share a bearer key (see.env.example):
# .env WATERMARKS_SYNTHID_SCORER_URL=http://wr-synthid-score:8766 WATERMARKS_SYNTHID_SCORER_API_KEY=change-me docker compose --profile heavy up -d
ThenPOST /cleanwith{"options": {"detect_before": true, "detect_after": true}}returnssynthid_before/synthid_afterin the report, andPOST /detecton an image returns the SynthID score. Fail-soft: if the sidecar is down or unconfigured, reports carry{"available": false, "error": ...}and cleaning still succeeds.
V4 scoring usesartifacts/spectral_codebook_v4.npzfrom the upstream checkout (220 MB). This isdetection/scoring only— it does not remove pixel watermarks.
Forpixel-domainimage watermarks (SynthID-class, StegaStamp, Tree-Ring, StableSignature), an optional external backend runs the CtrlRegen pipeline (ControlNet + DINOv2 IP-Adapter controllable regeneration). The backend ismertizci/noai-watermark, a maintained reimplementation of the ICLR 2025CtrlRegenmethod with automatic tiling.
The backend isnot bundledand ships no LICENSE file, so it is treated as all-rights-reserved: it is cloned at a pinned commit and loaded at runtime. Its research-era dependency pins (requirements-ctrlregen.txt— e.g.transformers==4.37.2,diffusers==0.27.2) carry published advisories and are intentionally not current, so they are only ever installed inside the dedicated venv this script creates and never into the main service image;setup_ctrlregen.shalso re-verifies the pinned commit on existing checkouts, not just fresh clones.
SCRIPTS=service/scripts # Clones upstream (pinned commit), creates a venv, installs torch + deps. "$SCRIPTS/setup_ctrlregen.sh" # Standalone removal (default checkout: ~/noai-watermark). NOAI_WATERMARK_DIR=~/noai-watermark \ ~/noai-watermark/.venv/bin/python "$SCRIPTS/clean_ctrlregen.py" shot.png -o shot.ctrlregen.png
On Windows usesetup_ctrlregen.ps1(same flags as-Dir,-Ref,-Python); the venv lands in.venv\Scripts\, whichclean_image.pyalready resolves. It probes the published PyTorch wheel indices and picks the highest one at or below the CUDA versionnvidia-smiprints that actually exists — that number is the maximum thedriversupports, and drivers are backward compatible, so a driver reporting 13.1 (no publishedcu131) installscu130. Below compute capability 7.5 it forcescu126, the last index whose wheels still carry Maxwell/Pascal/Volta kernels. It installstorchandtorchvisiontogether from that index so the dependency install cannot swap them for CPU builds from PyPI, then verifies after install thattorch.cuda.is_available()is true — if a GPU was detected but torch ends up CPU-only, the script warns loudly and exits non-zero instead of pretending the setup succeeded.
NOAI_WATERMARK_DIR=~/noai-watermark \ ~/noai-watermark/.venv/bin/python "$SCRIPTS/clean_image.py" shot.png \ -o shot.cleaned.png --remove-pixel ctrlregen
Order of operations: metadata strip first, then CtrlRegen pixel removal, then an optional reverse-SynthID before/after score (whenREVERSE_SYNTHID_DIRis also set).
Strength is conservative by default(--ctrlregen-strength 0.25), because higher strength removes more watermark but regenerates more of the image. Documented presets:0.15minimal /0.25default /0.35balanced /0.5aggressive /0.7max (backend default is 0.5).--ctrlregen-stepsdefaults to 50 (effective denoising steps ≈ steps × strength).
CtrlRegen is a 512×512 Stable Diffusion 1.5 ControlNet. The backend resolves this for arbitrary inputs, so no extra tiling is exposed here:
- ≤512 px:single pass — center-crop/resize to 512, regenerate, resize back.
- >512 px:automatic overlapping tiling (512 px tiles, 192 px overlap), width/height aligned to multiples of 8, then cosine-blended seams.
- Either path:output is resized to the original size and color-matched to the original image.
Very large images (e.g. 4K) produce many tiles, so runs scale with tile count (slower and higher VRAM). Pre-downscale large inputs when practical; tile size and overlap are hardcoded upstream and are not exposed as flags.
Expect ~10 GB of model downloads; a GPU is strongly recommended and CPU runs are slow. Some upstream models are gated, so exportHF_TOKEN(env only — never argv).clean_ctrlregen.pyrefuses to auto-install dependencies; runsetup_ctrlregen.shfirst.
There is no local detector for StegaStamp/Tree-Ring/StableSignature, so the only local signal is the reverse-SynthID score (a surrogate). When available,clean_image.py --remove-pixel ctrlregenreports that score before/after; the official Google SynthID check remains the final authority.
make docker-ctrlregen-build docker run --rm -e HF_TOKEN="$HF_TOKEN" \ --user "$(id -u):$(id -g)" \ -v "$(pwd):/data" \ watermarks-remover-ctrlregen /data/shot.png -o /data/shot.ctrlregen.png
Optional MarkLLM text-watermark verification
Forcontrolled experiments, an optional external harness wrapsTHU-BPM/MarkLLM(Apache-2.0) to watermark test text and re-detect it after a Layer B rewrite — e.g. prove that a KGW (Kirchenbauer, your "open-LLM" row) or SynthID-Text (Gemini row) mark disappears under your rewrite. It is averification harness, not an oracle: MarkLLM detection is only valid against thesamescheme config + keys used at generation, and it cannot certify a vendor detector will fail.
The backend isnot bundled.setup_markllm.shclones upstream at a pinned commit, creates a venv, and installs pinned deps (torch + transformers); the scoring model (defaultfacebook/opt-1.3b, Apache-2.0) downloads from Hugging Face on first run.
SCRIPTS=service/scripts # Bootstrap (clones upstream, creates ~/MarkLLM/.venv, installs deps). "$SCRIPTS/setup_markllm.sh" # Generate watermarked + unwatermarked sample text under the KGW scheme. MARKLLM_DIR=~/MarkLLM \ ~/MarkLLM/.venv/bin/python "$SCRIPTS/detect_text_watermark.py" watermark prompt.txt \ --scheme kgw -o wm.txt -o2 plain.txt # Detect the scheme mark in a text file. MARKLLM_DIR=~/MarkLLM \ ~/MarkLLM/.venv/bin/python "$SCRIPTS/detect_text_watermark.py" detect wm.txt --scheme kgw --json
Verification around a Layer B rewrite:pass--markllm-schemetorewrite_text.py(with--markllm-dir), and it records the MarkLLM detection before/after plus acleared`flag:
Sign in to leave a review
Use Google, GitHub, or an email account so ratings stay tied to real people.
No reviews posted yet.
