AI Roguelite

AI Roguelite

评价数不足
Flux Schnell GGUF via ComfyUI
   
奖励
收藏
已收藏
取消收藏
文件大小
发表于
117.588 KB
2024 年 12 月 25 日 上午 10:31
1 项改动说明 ( 查看 )

订阅以下载
Flux Schnell GGUF via ComfyUI

描述
Supports generating game images with Flux Schnell GGUFs using ComfyUI.
What is all this?
FLUX.1 is a family of image generation models by BlackForestLabs, which have similar natural language parsing capacity as Dall-E and some of which are openly available. Specifically, Flux Schnell is a freely available model designed to give good outcomes in as little as four steps. Moreover, Flux models can be quantized with the GGUF format to massively reduce their size while still having good output, so a good-ish gaming pc can actually run a Flux Schnell GGUF locally.
ComfyUI is a highly customizable image-generation backend supported by the game, and this mod will let you use Flux Schnell with the game.

Setup
It's not as easy as just installing the mod. Take the following steps to prepare:
  1. Install ComfyUI[github.com]
  2. Install the ComfyUI GGUF plugin[github.com]
  3. Get one of the Flux Schnell GGUF files[huggingface.co] and put it in the appropriate place for ComfyUI to find (it goes under models/unet). Pick a file that will fit entirely in your video card's VRAM. Generally bigger is better, but the Q8 may be overkill and the Q2 may be too low-quality to bother.
  4. Get clip_l.safetensors and t5xxl_fp16_em3fn.safetensors from this repository[huggingface.co]. They go under models/clip in ComfyUI.
  5. Get ae.safetensors from the main Flux Schnell repo[huggingface.co]. It goes under models/vae in ComfyUI.
Testing
Start ComfyUI. Click the image below, then drag it from your browser (or download) onto ComfyUI's main window to obtain the test workflow. (Due to Steam safety precautions dragging it directly won't work, nor can it be dragged from the screenshots.) The workflow should look a lot like in the screenshot above.
[i.postimg.cc]
In the top-left change unet_name to match the GGUF you downloaded. If everything went correctly, this should be the only needed change. (The same flow will be used by the mod and most of it will not be changeable in-game, so you must get things to a point where it works changing only unet_name.) Queue the prompt and wait. The generated image should be the same or almost the same as the sample image. (I generated with the Q5 myself, there may be slight differences with other quantizations.)

Usage
Have the mod installed and run ComfyUI before running the game. The connection with ComfyUI can be fragile and loading the model and clip can take significant time, so I very much recommend first generating an image with Flux Schnell yourself so it's loaded properly before you run the game.
In-game, make sure the mod is active and choose ComfyUI in Image Gen settings. Under "CKPT file name", fill in exactly what worked for you under unet_name in the test. Reduce steps down to 4. Flux is mostly trained on 1024x1024 images and should do those quickly enough, but its accuracy doesn't degrade on lower dimensions like with Stable Diffusion so you can pick a smaller size than that (as on most monitors the game won't render them that big anyway). Bear in mind that Flux does not normally use a negative prompt so what you fill in for that will be ignored.
Other models
Flux is harder to make finetunes and custom models for so there isn't the same cottage industry of custom models as for SD1.5, but there are some. If you want, you can download a custom model GGUF from Civitai[civitai.com] and use that instead. As with the main Flux Schnell GGUF, you'll have to make sure it works in a test and set the model name to match in settings.
3 条留言
XXTOXICHAIL 3 月 24 日 上午 8:19 
godsend. thanks mate
Harald B  [作者] 2024 年 12 月 28 日 上午 3:01 
Glad to hear it. I was actually hesitating to publish this; I wasn't sure it would add much since it seemed like making the mod was easy compared to getting Flux GGUFs to run to begin with.
Hunger 2024 年 12 月 27 日 下午 1:06 
Thanks for this, I could never get Flux to work with this game because I didn't know what I needed. This makes it take only like 10 minutes to setup