ziyue-kexin-jieyu

[ziyue-kexin-jieyu]

E. Integrate Scene Models into the City Model Based on Scene Keywords and GIS Location Points

Using the scene-related keywords from the ten groups, corresponding location points in the GIS data are selected to place the generated scene models into the urban model.

[ziyue-kexin-jieyu]

D.Scene Model Generation

Method 2: Image → Element Images → Reconstructed Image

artworks（group1-group10）

films（group1-group10）

ideal home show（group1-group10）

a. Select element images from those corresponding to 10 groups of sentence sets (images are sourced from the previously collected three types of image datasets). b. Based on sentence definitions, recompose the element images (more than 10) into a single spatial image. c. Read multiple .png images from a specified folder, randomly select more than 10 images each time, scale them proportionally, and paste them at random positions onto a canvas. Save the final result as a new collage. A total of 30 such collage images will be generated.

Use Stable Diffusion + Local Masking (Inpainting) to modify only local regions of the collage. Each image retains its original structure. Using HuggingFace's diffusers, apply a mask to each image and generate new content in that masked area based on the sentence.

Tools and Libraries:

numpy.random: Used for random image selection and random placement.

Pillow (PIL): Image processing (reading, scaling, pasting, saving, etc.).

numpy: Matrix operations for coordinates, masks, etc.

diffusers (from HuggingFace): For loading and running the Stable Diffusion model (specifically the inpainting model).

torch: Runs the model and handles tensors.

transformers: Used for handling prompts or interacting with the model.

cv2 or PIL: Image mask processing.

matplotlib.pyplot: Can be used during debugging to display images.

[ziyue-kexin-jieyu]

D. Scene Model Generation

Method 1: Image → Element Image → 3D Model

a. Select element images from the image sets corresponding to 10 groups of sentences (the images are sourced from the previously collected three categories of image datasets).

b. Generate 3D models of elements from the element images:

OpenCV: Image processing, contour extraction

Shapely: Polygon processing

Trimesh: 3D modeling and texture mapping

Pillow: Texture processing

NumPy: Numerical computation

c. Generate scene models based on sentences and element models:

Trimesh: Loading, composing, transforming, and exporting 3D models

Random: Random generation

NumPy: Numerical calculation

Variable settings:

Number of spatial hierarchy levels

Number of element models included

Number and hierarchy level of randomly duplicated element models

Rotation, scaling, and translation values of element models in space

[ziyue-kexin-jieyu]

C. Extract relevant elements from images based on a list of element keywords and associate them with corresponding sentence groups.

artworks

ideal home show

films

Semantic segmentation model: CLIP-seg + YOLO.

[ziyue-kexin-jieyu]

B. Use keywords to extract relevant sentences from the text and match the sentences with images.

key words to sentences

sentences-images

10 random group：

G1:['sorrow'，‘hallway'，‘airport"]

G2:['anxious'，'storage room'，‘market']

G3:['excited'，'corridor'，‘park']

G4:['happy'，‘bedroom'，'seaside'］

G5:['sad'，'study'，‘hospital']

G6:['safety'，‘balcony'，‘historicalsite'］

G7:['calmness'，‘basement'，'station']

G8:「'anxious'，'corner'，‘bus']

G9:['depressed'，‘foyer'，‘shopping mall'］

G10:［'belonging'，‘home'，'street']

Keywords to Sentences: TF-IDF Sentence to Image Matching: CLIP

[ziyue-kexin-jieyu]

A. Collecting Home-Related Content (Texts, Images, and Videos)

Content to be collected includes texts (from magazines, novels, academic literature), images (e.g., from the Ideal Home Show, artworks), and videos.

over 1000 images about "Artworks"

over 600 films about "Home"

over 3000 images about "Ideal Home Show"

500 magazines, novels, academic literature about " Home "

Scraping Tool: BeautifulSoup Key word: Home Sources:

Literature: https://artvee.com/, https://scholar.google.co.uk/

Images: https://artvee.com/

Videos: https://panoramasofcinema.ch/

[ziyue-kexin-jieyu]

4.Space Generation System

A. Crawl texts related to "home" (magazines, novels, literature), as well as images (Ideal Home Show, artworks) and videos. B. Use keywords to extract relevant sentences from the texts, and match these sentences with images. C. Extract relevant elements from the images based on a list of element keywords and associate them with the corresponding sentence groups. D. Generate scene models. E. Place the models into an urban model based on keywords.

[ziyue-kexin-jieyu]

3.Design Objective

To introduce emotionally-driven 'home' spaces into vacant properties across different parts of the city, serving as anchors to fill urban voids and reconnect people with space.

[ziyue-kexin-jieyu]

2.Starting Point of the Proposal – 'Home'

'Home': A word that embodies both emotion and space.

Inspiration: the 'Ideal Home Show' and psychological tests about the 'ideal home'（https://www.idealhomeshow.co.uk/）.

[ziyue-kexin-jieyu]

1.Urban research

In Athens, there are many vacant houses, and the levels of activity vary significantly across different areas. Moreover, there is a noticeable contrast in the appearance and living environment between the old and new parts of the city.

[ziyue-kexin-jieyu]

IMAGES

2.YOLO+DEEPLAB

file:///E:/TERM%202/ideal_home_show/image2_output-som/som_bmu_visualization.html

[ziyue-kexin-jieyu]

IMAGES

2.YOLO+DEEPLAB

file:///E:/TERM%202/art/elements_output-som/som_bmu_visualization.html

[ziyue-kexin-jieyu]

IMAGES

NumbaSOM

file:///E:/TERM%202/ideal_home_show/main-som/som_bmu_visualization.html

[ziyue-kexin-jieyu]

IMAGES

1.NumbaSOM

file:///E:/TERM%202/art/main-som/som_bmu_visualization.html

[ziyue-kexin-jieyu]

TEXT

text-clip-images

[ziyue-kexin-jieyu]

[ziyue-kexin-jieyu]

One of the groups where the description texts of the two images are most similar.